Thin wafer-level camera lenses inspired by insect ... - OSA Publishing

Thin wafer-level camera lenses inspired by insect compound eyes 1 Jacques Duparr´ ¨ Andreas Bruckner, e,1,3 Robert Leitel,1 1,2 ¨ Peter Dannberg,1 Andreas Bräuer,1 and Andreas Tunnermann 1 Fraunhofer

Institute for Applied Optics and Precision Engineering, Albert-Einstein-Str. 7, D-07745 Jena, Germany 2 Friedrich Schiller University Jena, Institute of Applied Physics, Jena, Germany 3 Now with Pelican Imaging Corp., 779 East Evelyn Ave. Mountain View, California 94041, USA *[email protected] http://www.microoptics.org

Abstract: We propose a microoptical approach to ultra-compact optics for real-time vision systems that are inspired by the compound eyes of insects. The demonstrated module achieves approx. VGA resolution with a total track length of 1.4 mm which is about two times shorter than comparable single-aperture optics on images sensors of the same pixel pitch. The partial images that are separately recorded in different optical channels are stitched together to form a final image of the whole field of view by means of image processing. A software correction is applied to each partial image so that the final image is made free of distortion. The microlens arrays are realized by state of the art microoptical fabrication techniques on wafer-level which are suitable for a potential application in high volume e.g. for consumer electronic products. © 2010 Optical Society of America OCIS codes: (110.0110) Imaging systems; (350.3950) Micro-optics; (230.3990) Micro-optical devices; (040.1240) Arrays; (330.1720) Color vision; (100.2000) Digital image processing; (150.0150) Machine vision; (220.0220) Optical design and fabrication.

References and links 1. S. Mathur, M. Okincha, and M. Walters, “What Camera Manufacturers Want,” 2007 International Image Sensor Workshop, Ogunquit Maine, USA, June 7.–10., 2007. 2. Z. Popovic, R. Sprague, and G. Connell, “Technique for monolithic fabrication of microlens arrays,” Appl. Opt. 27, 1281–1284 (1988). 3. D. Daly, R. F. Stevens, M. C. Hutley, and N. Davies, “The manufacture of microlenses by melting photoresist,” Meas. Sci. Technol. 1, 759–766 (1990). 4. J. Duparré, P. Dannberg, P. Schreiber, A. Bräuer, and A. Tünnermann, “Thin compound-eye camera,” Appl. Opt. 44, 2949–2956 (2005). 5. A. Brückner, J. Duparré, F. Wippermann, P. Dannberg, and A. Bräuer, “Microoptical Artificial Compound Eyes,” in Flying Insects and Robots, D. Floreano, J.-C. Zufferey, M. V. Srinivasan and C. Ellington, eds., (Springer, 2009), pp. 127–142. 6. A. Brückner, J. Duparré, P. Dannberg, A. Bräuer, and A. Tünnermann, “Artificial neural superposition eye,” Opt. Express 15, 11922–11933 (2007). 7. E. Buschbeck, B. Ehmer, and R. Hoy, “Chunk versus Point Sampling: Visual Imaging in a Small Insect,” Science 286, 1178–1179 (1999). 8. W. Pix, J. M. Zanker, and J. Zeil, “The Optomotor Response and Spatial Resolution of the Visual System in Male Xenos Vesparum (Strepsiptera),” J. Exp. Biol. 203, 3397–3409 (2000).

#133960 - $15.00 USD

(C) 2010 OSA

Received 7 Sep 2010; revised 25 Oct 2010; accepted 27 Oct 2010; published 8 Nov 2010

22 November 2010 / Vol. 18, No. 24 / OPTICS EXPRESS 24379

9. M. Shankar, R. Willett, N. Pitsianis, T. Schulz, R. Gibbons, R. T. Kolste, J. Carriere, C. Chen, D. Prather, and D. Brady, “Thin infrared imaging systems through multichannel sampling,” Appl. Opt. 47, B1–B10 (2008). 10. Y. Kitamura, R. Shogenji, K. Yamada, S. Miyatake, M. Miyamoto, T. Morimoto, Y. Masaki, N Kondou, D. Miyazaki, J. Tanida, and Y. Ichioka, “Reconstruction of a High-Resolution Image on a Compound-Eye ImageCapturing System,” Appl. Opt. 43, 1719–1727 (2004). 11. K. Nitta, R. Shogenji, S. Miyatake, and J. Tanida, “Image reconstruction for thin observation module by bound optics by using the iterative backprojection method,” Appl. Opt. 45, 2893–2900 (2006). 12. R. Horisaki, S. Irie, Y. Nakao, Y. Ogura, T. Toyoda, Y. Masaki, and J. Tanida, “3D information acquisition using a compound imaging system,” Proc. SPIE 6695, 66950F (2007). 13. G. Druart, N. Guérineau, R. Haidar, S. Thétas, J. Taboury, S. Rommeluère, J. Primot, and M. Fendler, “Demonstration of an infrared microcamera inspired by Xenos peckii vision,” Appl. Opt. 48, 3368–3374 (2009). 14. T. Chen, P. Catrysse, A. E. Gamal, and B. Wandell, “How small should pixel size be?” Proc. SPIE 3965, 451–459 (2000). 15. S. Farsiu, D. Robinson, M. Elad, and P. Milanfar, “Advances and challenges in super-resolution,” Int. J. Imaging Syst. Technol. 14, 47–57 (2004). 16. A. Lohmann, “Scaling laws for lens systems,” Appl. Opt. 28, 4996–4998 (1989). 17. R. Kinglake, A History of the Photographic Lens (Academic Press, 1989). 18. J. Duparré, F. Wippermann, P. Dannberg, and A. Reimann, “Chirped arrays of refractive ellipsoidal microlenses for aberration correction under oblique incidence,” Opt. Express 13, 10539–10551 (2005). 19. P. Dannberg, L. Erdmann, R. Bierbaum, A. Krehl, A. Bräuer, and E. B. Kley, “Micro-optical elements and their integration to glass and optoelectronic wafers,” Microsyst. Technol. 6, 41–47 (1999). 20. P. Dannberg, F. Wippermann, A. Brückner, A. Matthes, P. Schreiber, and A. Bräuer, Fraunhofer Institute for Applied Optics and Precision Engineering, Albert-Einstein-Str. 7, D-07745 Jena, Germany, are preparing a manuscript to be called “Wafer-Level hybrid integration of complex micro-optical modules.”

1.

Introduction

Ongoing developments in the semiconductor industry lead to shrinking image sensor formats for miniature digital cameras which are driven by a cost-cutting need for integrated optoelectronics. Many consumer applications but also medical imaging and safety devices profit from this trend. Hence, miniature cameras are now integral parts of portable information devices like PDAs, laptops and mobile phones, but also chip-on-the-tip endoscopes and automotive sensors. Besides the reduction of fabrication costs, a smaller image sensor format allows to reduce the dimensions of the lens so that the overall camera size could be decreased. However, what seems to be a simple scaling procedure causes some delicate difficulties. Injection molding is still the dominating fabrication technology for the lens fabrication of today’s miniature camera modules. The need for highly precise alignment of components during the barrel integration severly increases the costs for lens diameters below 4 mm. This is intensified when considering the larger resolution in image space which is demanded by the shrinking pixel size, leading to insufficient performance with the standard fabrication techniques [1]. During the last years, an alternative fabrication technique which involves UV-molding, turned out to be a promising candidate for the fabrication of small lenses because the fabrication and their integration with spacers is carried out in a wafer-level process for thousands of lenses in parallel. Existing solutions use complex aspherical lens profiles whose realization is limited due to problems with mold mastering, shrinkage during the UV molding process as well as meeting the tolerances for all of the vertical layers within a single module. Nevertheless, millions of these single-aperture wafer-level optics (WLO) modules are sold per year to be used mainly as secondary camera in mobile phones. The discussed issues put a limitation on the performance and yield of WLO cameras so far. The development which is described here uses an alternative approach for the miniaturized optical system itself which is known from the insect compound eye. We propose a multiaperture imaging system where separate optical channels capture different portions of the overall field of view. This segmentation changes the trade-off between focal length (which is directly related to the minimum total track length) and size of the field of view so that a short total track

#133960 - $15.00 USD

(C) 2010 OSA



length may be achieved at the same time with using a small field of view for each individual channel. The miniaturization of the individual optical channels allows to use microlens arrays which, in contrast to single-aperture wafer-level optics, have small lens sags and suffer less from shrinkage during molding. Additionally, master generation is carried out by means of photolithography and reflow of photoresist both on wafer-level [2, 3]. 2.

Prior work and biological inspiration

During the last decade, several miniaturized multi-aperture imaging systems have already been demonstrated. For instance, the so-called artificial apposition compound eye uses one pixel per microlens [4]. The latest version is characterized by a very large field of view (FOV) of 92◦ and a thickness of 0.3 mm. On the other hand, it consumes a large sensor footprint and has a low resolution of about 150 x 100 pixels restricting its use to sensing applications [5]. An improvement of the performance of artificial apposition compound eyes is bionically inspired by the eye of the house fly (lat.: Musca). The artificial neural superposition compound eye uses a set of 3x3 pixels in the footprint of each microlens. Color imaging and an increase of the signal to noise ratio have been demonstrated using a redundant sampling between these pixels [6]. However, the main disadvantage of the apposition principle, the low image resolution, remains. Recently, different groups came up with the idea to read out the complete partial image in each channel of a multi-aperture imaging system and use image processing to fuse the different partial images in order to form an overall image of higher resolution. The working principle is inspired by a wasp parasite called Xenos Peckii. It has been reported that males of Xenos Peckii bear a very particular kind of compound eye which reveals an extended retina at the image plane of the cornea lens of each isolated ommatidium [7]. Further anatomical features hint at a cross correlation between individual retina blocks of adjacent ommatidia of the compound eye in order to improve its spatial resolution [see Fig. 1(a)]. Although other physiologists are still in doubt about whether the eye of Xenos Peckii is an evolutionary optimum or an intermediate state [8], for technical solutions, it seems beneficial to combine such a setup with methods of super-resolution.

Fig. 1. (a) Schematic working principle of the compound eye of Xenos Peckii (after [7]). Different extended parts of the object space (illustrated by letter sequence) are imaged onto the retina of each individual eyelet. The signals of each retina are rotated in the early neural layers. Finally, the partial images are linked up to a real image of the object space in the lamina. (b) Schematic section demonstrating the working principle of an electronic cluster eye (eCley).

Two basic schemes can be distinguished here: (1) In each channel, a low resolution image of the full FOV is captured. An image registration and fusion algorithm is used to create the final image from all partial images. This approach is #133960 - $15.00 USD

(C) 2010 OSA



promising for creating thin and inexpensive imaging optics for the infrared spectral range [9] or for 3D imaging of close objects [10–12]. The small FOV, which is limited to 10-22◦ by the numerical aperture of the optics in each individual channel, is an inherent problem of the proposed implementations. Additionally, the object distance is limited to a range between a few centimeters up to approx. two meters in order to produce sub-pixel displacements between neighboring partial images which is a necessary condition for the super-resolution to work. The processing time makes any real-time imaging impossible so far. (2) In each channel, only a part of the whole FOV is recorded and a final image is created by the stitching of all partial images by means of software processing. This principle has been used to create a compact imaging system with a FOV of 30◦ in the infrared spectral range [13]. Our solution demonstrates the capability of the second scheme to achieve compact size together with a large FOV and good sensitivity in the visible spectral range. We increase the number of pixels per channel so that a cluster of small partial images is captured. The partial images are then stitched together electronically which lead to the name - electronic cluster eye (in the following abbreviated to eCley). 3.

Working principle of an electronic cluster eye

A simple geometric relationship [Eq. (1)] relates between the effective focal length ( fe f f ), the diameter of the image circle (I) and the FOV (represented by its half maximum angle of incidence α ) for a single-aperture camera lens. fe f f =

I . 2 tan (α )

(1)

Usually, for miniaturized camera lenses with a focal length of less than 5 mm the minimum total track length is about 1.1 to 1.5 times greater than the effective focal length due to the correction of aberrations and packaging issues. Telephoto designs are very rare at that scale. Let us assume that a certain FOV (2α ) and f-number (F/#) are given by the application. The image with a certain number of pixels (Nx , Ny ) is captured on an image sensor with given pixel pitch (p px ). For a distortion-free sampling of the FOV we calculate the sampling angles in x,y: tan αy tan αx ; Δφy = arctan . (2) Δφx = arctan Nx /2 Ny /2 Here, αx , αy denote the half cone angles of the FOV along the horizontal and vertical direction, respectively (Fig. 2). This may be expressed by the half angle of the full diagonal FOV (α ) and the ratio of the horizontal to the vertical edge length of the image using the following two equations: Nx tan αx = (3) tan αy Ny tan2 α = tan2 αx + tan2 αy .

(4)

On the other hand, in order to achieve a specific angular sampling we need to have a certain focal length ( fsa ) together with the pixel pitch (p px ): fsa =

p px , tan (Δφ )

(5)

where the sampling angle is constant in the vertical and horizontal direction Δφx = Δφy = Δφ . #133960 - $15.00 USD

(C) 2010 OSA

(6)



Fig. 2. Visualization of the basic parameters which characterize the FOV of a camera system. The number of image pixels (Nx , Ny ) directly correspond to the number of object sampling intervals. The object distance is denoted by OD.

Once the focal length is known we may easily derive the diameter of the aperture stop (D) by inverting F/# = f /D. Although these few equations are useful only for small angles of incidence (paraxial region) and they do not account for any aberrations, they already indicate the minimum size of the optics module. For example, if we want a short focal length, the image diagonal will have to decrease [Eq. (1)] and therefore the pixel pitch has to be decreased as well in order to keep a constant angular sampling [Eq. (5)] and constant number of pixels in the image. That means that the focal length of a conventional camera lens decreases mainly due to the reduction of the pixel size (assuming a fixed number of pixels and fixed FOV). This relationship is commonly exploited in the mobile phone camera market which reduces the dynamic range [14]. This trade-off can be modified by introducing multiple isolated optical channels with a certain pitch difference between their lenses and partial images: Δpk = pL − pk ,

(7)

where pL denotes the pitch of the microlenses and pk the pitch of the partial images. The segmentation of the total FOV by multi-aperture optics has two advantages when it comes to miniaturization: Firstly, it enables the use of an optical setup with low complexity for each channel because each transmits only a limited part of the FOV and aberrations do not have to be corrected for the full field per channel. A comparable single-aperture system would need an aberration correction over the entire field which tends to make the optical system more complex and therefore causes a larger total track length. Secondly, the viewing direction in the FOV may be chosen easily by applying the pitch difference (Δpk ). Hence, the angular sampling of the object space is not necessarily limited by the pixel pitch and the focal length of a single channel. Following the concept of the eye of Xenos Peckii, we designed a microoptical imaging system where each individual optical channel captures a partial image of a part of the overall FOV [see Fig. 1(b)]. The partial images are recorded by an image sensor array and are electronically stitched together to form a final image with higher resolution. In contrast to other super#133960 - $15.00 USD

(C) 2010 OSA



resolution algorithms which optimize the fusion of several low resolution images of the full FOV with unknown image registration [10, 15], we rather maintain a well defined relationship between the viewing directions of adjacent channels by thorough setting up the pitch difference between the optics and the partial image of each individual channel. The partial images of each adjacent pair of microlenses are braided (see Fig. 3).

Fig. 3. Principle of braided sampling of the object space by adjacent optical channels with k = 2. Colors are used for visualization only. Equal colored items refer to the same optical channel.

The reason for that is that the effective pitch difference (Δpe f f ) is designed in a way that the partial images of adjacent channels are displaced by a non-integer part of the pixel pitch p px and hence allow a smaller effective sampling angle (Δφe f f ) Δpe f f =

p px , k

Δpe f f p px tan Δφe f f = = . f k· f

(8) (9)

Here, the integer k ≥ 1 is the so-called braiding factor and f the focal length of a single optical channel. Otherwise, the sampling angle (Δφ ) in a single channel would increase when decreasing the focal length for a fixed pixel size [16]. Together with Eq. (5) this yields: tan Δφe f f fsa = . (10) tan (Δφ ) k· f The aim is to achieve a constant angular sampling for the multi-aperture system compared to the single-aperture system as this is desired for miniature camera applications. Hence, the condition fsa ! =1 (11) k· f

#133960 - $15.00 USD

(C) 2010 OSA



has to be met, which yields f=

fsa . k

(12)

The advantage of this setup is two-fold: • The effective focal length can be shortened by approx. 1/k while the angular sampling is kept constant. • The light sensitivity is increased because the object field is imaged using a shorter focal length on the same pixel size. However, we have to note that the fill factor of the area of read-out pixels compared to the total active area of the image sensor array is considerably lower than 100 percent in case of using a commercial image sensor. This is caused by unused intermediate space between microlenses that is necessary in order to achieve an acceptable sensitivity and for the integration of structures to prevent optical crosstalk. If we assume an odd number of pixels (Ng ) along one dimension of each partial image, then the pitch difference Δpk may be expressed by the following Ng · p px , (13) Δpk = k in order to hold for the proposed sampling scheme. Now, the ratio between the size of a single squared partial image (Ip = Ng · p px ) and the partial image pitch (pk ) gives the amount of used active image sensor area compared to the unused active area in case of a dense commercial image sensor array. We may write: Ng · p px . (14) Γ= pk In case of Γ = 1, 100% of the active area is used which, in our case, would lead to crosstalk between neighboring channels and thus low image contrast. We rather choose Γ ≤ 0.5 in order to achieve a sufficient suppression of crosstalk. The multi-aperture approach of eCleys asks for a specific image sensor layout with small groups of densely packed photodiodes. For the demonstration system (see next section) we were bound to commercial image sensors so that an image sensor array of larger resolution is needed for a certain resolution of the eCley. In that case, the scaling behavior of the needed image sensor format size and the Γ parameter depending on the image resolution of the eCley and the choice of number of pixels per channel can be derived from Fig. 4. 4.

Optical layout of an electronic cluster eye with VGA resolution

An optical design has been created for the demonstration of an eCley following the concepts discussed above. The optical setup of each channel is rather simple and contains a single microlens and an aperture stop (Fig. 5). Some optical aberrations like coma, astigmatism and field curvature are reduced by placing the aperture at a certain distance ahead of the lens - a modified setup of the well known reversed landscape lens [17]. Additionally, the reversed microlens array causes a higher telecentricity because ray bundles are refracted at the front face before hitting the individual microlens and the front face acts as a cover glass. From the scaling laws shown in Fig. 4 we find a suitable number of pixels per partial image (nx,y ) and the number of channels in x,y direction (Kx,y ) using Kx,y =

#133960 - $15.00 USD

(C) 2010 OSA

Nx,y . nx,y

(15)



Fig. 4. Lateral scaling behavior of the proposed eCley when using commercial image sensors. The fill factor Γ (ratio of partial image size and pitch of the partial images, black solid lines) as well as sensor format size in pixels along one dimension (red solid lines) are shown as functions of the final image resolution (x axis) and the number of pixels per channel (y axis). The bold black line marks the case of a maximum fill factor of Γ = 1. A fixed FOV of 70 degrees, a F/# of 2.8 and k = 2 are assumed.

Fig. 5. Schematic cross section of the optical layout of the electronic cluster eye prototype. The optical module consists of different layers and is directly attached to the image sensor (shown in gray). Glass substrates are indicated in light blue. Dark blue and green color is used for the Ormocomp material (Micro Resist Tech.) of the microlenses and the polymer spacer, respectively.

#133960 - $15.00 USD

(C) 2010 OSA



The optimization of each individual channel’s properties and the simulation of the system performance has been carried out with the commercial raytrace software ZEMAX. The central viewing direction of each channel is determined by the pitch difference between the microlens array and the partial image ΔpK . It is designed in a way that the central viewing directions of all channels form a grid with equidistant spacings on a distant object plane - in other words: the central image points of all channels are free of optical distortion. The size of the partial image (Ip ) and the focal length ( f ) define the FOV of the individual channel. The FOV in each channel is kept at about 10 degrees which enables the application of microlenses with low sags. Additionally, we tune (’chirp’) the tangential and sagittal radii of curvature of each microlens according to its individual viewing direction, in order to compensate for astigmatism and field curvature within each partial image [18]. Some basic system parameters of the current optical design are listed in Table 1. Table 1. Parameters of the VGA eCley demonstration systems

Property size of optics module (L x W x T) [mm] field of view (H x V) f-number final image resolution [pixels] no. of channels partial image size [pixels] pixel pitch [μ m] diameter of microlens* [mm] radius of curvature* [mm] ∗ Denotes

Value 6.8 x 5.2 x 1.4 58◦ x 46◦ 3.7 700 x 550 17 x 13 39 x 39 3.2 0.375...0.386 -0.492...-0.531

the minimum and maximum values in the chirped microlens array.

The modulation transfer functions (MTF) as well as the focal spot sizes have been analyzed at several angles of incidence in order to make predictions about the resolution. MTF versus focus plots were used to calculate the depth of focus and image simulations were made to simulate distortion within selected single channels. As optical crosstalk is a severe problem for multichannel imaging systems, it has to be strictly avoided. Otherwise, it would cause ghost images due to light that is imaged by one microlens onto the photodiodes of a neighboring channel. In order to suppress optical crosstalk, adjacent optical channels have to be optically isolated e.g. by opaque walls or multiple horizontal diaphragm arrays. An aperture array at the front end of the optical system acts as absorber for light which would otherwise hit the intermediate space between lenses and therefore would cause stray light and reduce the image contrast. Three additional diaphragm arrays are crucial to suppress the possible forms of optical crosstalk in the present prototype. We added these absorbing diaphragm arrays to our optical design model and optimized their position and the size of their openings for maximum suppression of crosstalk. A non-sequential raytracing analysis has been carried out for that purpose, so that the simulation accounts for object distance, reflections, stray light and crosstalk. An analysis of the veiling glare index has been used to quantify the influence of crosstalk and stray light on the final image. Image simulations have been made in order to test the image stitching and distortion correction algorithms. Figure 6(a) and 6(b) demonstrate some of the simulation results.

#133960 - $15.00 USD

(C) 2010 OSA



Fig. 6. Simulated images of the eCley using a non-sequential ray tracing method and a twodimensional representation with lambertian illumination for the target object. (a): CCTV test chart. (b): An image of an array of spoke targets. The image resolution is 700 x 550 pixels in both cases.

5.

Technological fabrication and integration

The optical setup of the VGA eCley exhibits very small lens sags in the range of 40 microns which makes the microlens array a perfect candidate for a fabrication with photolithography, reflow and subsequent UV-molding for replication. The fabrication of such microoptical modules is carried out in multiple steps which are listed in Table 2. Table 2. Fabrication steps and their achieved accuracy

Step (1) Structural data export and binary mask design (2) Lithography mask fabrication (3) Generation of master wafers for microlens arrays (4) Creation of replication tools (5) Preparation of glass wafers and diaphragm arrays (6) Wafer lamination (7) UV-molding of microlenses (8) UV-molding of lens spacers (9) Integration of lens wafer and spacer wafer (10) Dicing of wafer (11) Alignment and packaging with image sensor

Comment on accuracy numerical precision of data e-beam lithography not critical not critical interlayer alignment lateral ±1μ m across 200mm not critical lateral ±1μ m across 200mm, axial ±5μ m, variation of ROC < 1% lateral ±1μ m, axial ±5μ m lateral ±1μ m, axial ±5μ m, full back focal length control within ±10μ m not critical lateral < 2μ m, axial ±10μ m, rotation ¡0.15 deg., tilt between planes ¡0.18 deg. (10μ m elevation)

The fabrication of master structures for replication involves UV-lithography in photoresist

#133960 - $15.00 USD

(C) 2010 OSA



in order to create binary structures and a subsequent reflow of the resist structures to create microlenses [2, 3]. After that, a UV transparent mold is replicated from the master in a soft elastomer material on a glass carrier substrate. Meanwhile, diaphragm arrays are structured on the glass wafers in a black-matrix polymer resist material (PSK 2000, Brewer Science) using again UV-lithography. Further details about these fabrication techniques may be found in [19,20]. Two glass wafers are sequentially laminated on a third one (with structured diaphragm layers on front and back sides) to form the spacer wafer with a total thickness of 1.1 mm. Another diaphragm array is created on the front side of the stack after an alignment step. The microlenses are molded in UV-polymer (ORMOCOMP, Micro Resist Technologies) on the second pre-processed glass wafer with a thickness of 0.21 mm. In a subsequent UV-molding step, the polymer spacers with a thickness of 20 μ m are created on top of the microlens arrays. This is necessary to achieve an air gap between the microlenses and the glass spacer wafer when the lens wafer is bonded upside down to the latter. Up to this stage, the whole fabrication process is a wafer-level technology which is highly precise and cost-efficient especially for a potential high volume production. Finally, the wafer is diced [see Fig. 7(a)] and single optics chips are aligned to the image sensor arrays. For our VGA demonstration system we used a 3MP image sensor on a demoboard. The cover glass of the sensor has been removed in order to directly access the active pixel matrix. We aligned the optics relative to the image sensor array while monitoring the focus distance along the z-axis (normal on the sensor array), rotation around the z-axis and tilt between both planes within the read-out of all partial images from the sensor. When the alignment was adequate, we adhered the microoptical module to the image sensor array [see Fig. 7(b)]. The package was later sealed with a flat cover hood to avoid light entering from the side faces.

Fig. 7. (a): Diced wafer with several optics modules of electronic cluster eyes. (b): Photograph showing the size comparison between the eCley with VGA resolution on the image sensor array and one cent coin.

6.

Experimental results and protoype

After assembly, the demonstration system has been tested for various properties. This section summarizes essential specifications which have been measured for the device and are going to be compared to the results from the simulation of the eCley.

#133960 - $15.00 USD

(C) 2010 OSA



6.1.

Image quality

We have to take a closer look at the image processing pipeline in order to understand the factors which influence the image quality in the eCley. For the current prototype, the image processing steps are: 1. Bayer-demosaicing and partially color correction done by the commercial read-out software 2. extraction of the partial image from each individual channel out of the overall image matrix (Fig. 8 (a)) 3. channel-wise bilinear correction of partial image distortion according to a calibration file derived from optical design 4. channel-wise linear interpolation of undistorted image pixels on regular pixel grid 5. mapping of the undistorted pixels of each partial image into the final image matrix including horizontal and vertical flipping as well as braiding Please note that the steps (2), (3) and (4) may be carried out for each channel in parallel, if this is implemented in the hard- or software algorithm in order to speed up the image acquisition. The current implementation of the braiding algorithm (5) is a very simple one: each individual pixel is mapped to a different, known position in the final image matrix which resembles its original neighborhood in object space according to Fig. 3. The processing is done in a simple loop as part of the image read-out and display software because the mapping behavior is deterministic. The process duration for the distortion correction, interpolation and image fusion was 20 ms on an Intel CoreDuo machine with 2.66 GHz which makes it feasible to be applied in real-time. Processing and capturing frames could easily be interlaced so that framerates of ¿ 30 fps may be achieved on standard hardware. The image captured by the demo-board as well as the final (processed) image of the eCley are shown in Fig. 8(a) and 8(b). The final image resembles the simulated image quite well except to some edge displacements (’zipper artefacts’) and color aliasing. The edge displacements which appear at junctions between adjacent channels are thought to be due to imperfections of the distortion correction and braiding algorithm as well as residual misalignments between the microlens and image sensor arrays. Here, especially rotation and tilt between both components are critical as they influence the pitch and central viewing directions across optical channels. For future work we plan to modify the distortion correction and stitching algorithm in a way that it is more versatile when it comes to alignment errors in a real system. The apparent color aliasing effects are caused by the demosaicing which is carried out before the image stitching. Hence, different single color signals on a large sampling grid in object space are mixed by the conventional demosaicing method of the demo-board. However, as the signals which belong to closer adjacent points in object space may only be found in adjacent pixels of the final image, it is likely to be decreased when the stitching of partial images is carried out before the demosaicing in the final image. Further software modifications are necessary to implement such order in the processing pipeline. We also tested the suppression of optical crosstalk using a collimated HeNe-LASER source in front of the eCley prototype which was mounted on a goniometer in order to change the angle of incidence for the illumination. First ghost images appeared for large integration times so that a suppression factor between ghost image and signal of about 1:20.000 has been determined. Besides the well-defined imaging situation in the lab, we captured images outside on a bright sunny day [see Fig. 9(a)] which demonstrate the image quality.

#133960 - $15.00 USD

(C) 2010 OSA



Fig. 8. Image of a CCTV test chart which has been acquired by the prototype of the eCley. (a): Image as it is recorded by the image sensor array with a full resolution of 3MP. The inset shows a magnified section of the 17x13 partial images. (b): Final image after the application of the image processing pipeline. The image resolution is 700x550 pixels.

Fig. 9. (a) View on the Fraunhofer-Instiute IOF as it has been captured with the eCley prototype system. The image resolution is 700x550 pixels. (b) Comparison between different polychromatic on-axis MTF curves. Solid red line: simulated MTF. Black squares: MTF measured within the center of the image plane of the optical module with relay lens. Green circles: MTF measured within the unprocessed image of the central channel of the fully assembled prototype. Blue triangles: MTF measured in the center of the final image of the prototype (optics + image sensor) including all image processing. The gray dashed line shows the on-axis MTF of a commercial WLO camera.

#133960 - $15.00 USD

(C) 2010 OSA



6.2.

MTF performance

The modulation transfer function (MTF) was measured using a slanted edge target (according to ISO 12233) in order to quantify the resolution of the eCley prototype. The results are shown in Fig. 9(b). We measured several different MTF curves in order to gain as much insight as possible in the sources of image degradation. The simulated on-axis MTF curve (shown as red solid line) was derived from ray-tracing software for zero degree angle of incidence using a calculation method which accounts for diffraction. The single channel MTF for the separate optics module (black squares) was measured with a 40x microscope objective which relayed the image plane of the microoptical module onto a CCD camera chip. This setup was used to magnify the partial image of an edge object centered in the FOV that was created by a single optical channel in the center of the eCley. The other two MTF curves were measured for the assembled prototype with the optics mounted to the CMOS image sensor. We captured an image of a slanted edge object in the center of the FOV and analyzed it in the central channel of the bare recorded image (green circles). Subsequently, the MTF of the final image (blue triangles) was measured using the edge image that resulted after the fusion and distortion correction algorithm of all partial images. The Nyquist frequency is 312 cycles per millimeter which is the effective pixel pitch in the final image due to the braiding of partial images. However, the physical Nyquist frequency of the image sensor is 156 cycles per millimeter. The measured single channel MTF of the optics module resembles the simulated MTF very well. There is a systematic offset of about 10 % which is caused by fabrication tolerances and residual defocus due to an incorrect spacer thickness. A significant reduction (by up to 25 %) of the total MTF is caused during the image acquisition by the image sensor. The reason is the limited MTF of the image sensor pixels as well as a further reduction of spatial resolution within the single color planes due to the Bayer color filter mosaic. The MTF performance is further decreased by at maximum 10-15 % due to the partial image distortion correction and stitching algorithm. Although the fabricated optics module has the ability to transfer spatial frequencies higher than half of the sampling Nyquist frequency, the MTF of the final image vanishes at about the cut-off frequency of the image sensor (approx. 160 cycles/mm) leading to aliasing effects [see Fig. 8(b)]. Another MTF curve (gray dashed line) of a commercial, single-aperture WLO camera (Omnivision Camera Cube Model OVM7690) has been added to Fig. 9(b) in order to compare our results to the state of the art. It shows a 10 % advance at mid spatial frequencies over the MTF of the final (braided) image of the eCley. However, the images of the WLO camera contain considerably more noise due to the small pixel size of 1.75 μ m. We assume that most of the degradation is caused by the image processing, namely the distortion correction as well as the stitching of partial images of the eCley. Hence, we are going to improve these algorithms within future work. Additionally, other ways to improve the resolution via image processing (e.g. deconvolution) are planned to be investigated for the eCley. 6.3.

Distortion and relative illumination

Two different types of distortion can be distinguished in the eCley: (1) Distortion which appears if the central viewing directions of the individual optical channels do not follow Eq. (2) and (2) distortion of the individual partial images. The first has been corrected by a non-constant pitch difference between microlens and partial image, the latter by means of image processing. Insufficient distortion correction for the individual partial images is hard to quantify from the final images of the eCley. It causes displaced image details (zipper artefacts) which on the other hand might as well be caused by residual misalignment between the microoptical module and image sensor. We captured an image of a regular grid [see Fig. 10(a)] which had knots at each central viewing direction of the individual channels and lines which connect these knots. From

#133960 - $15.00 USD

(C) 2010 OSA



the image coordinates of the knots, we measured and verified the pitch of the partial images to be 111 pixels. The distance of each pair of grid lines was measured across the FOV [Fig. 10(b)]. A mean distance between adjacent lines of 38.87 pixels was measured and compared to the designed value (39 pixels) which gives a deviation of 0.3 %. Furthermore, no spatial distribution of the distortion over the final image was identified. Thus, the image of the eCley is nearly free of distortion.

Fig. 10. (a) Image of a square grating target. Color has been transformed to grayscale in order to measure distortion. The line scan for the grayscale profile in (b) is shown in the dashed black line. (b) Line scan through image of the grid. The measured center of each grid line is marked by a tick label on the x-axis in order to characterize distortion.

We additonally analyzed the relative illumination in image space from Fig. 10(b). Within the horizontal scan which corresponds to a full field angle of about 50◦ we measured a drop to about 69 % of the maximum illumination at the edge of field which is in good agreement with the simulation. 7.

Conclusion

The application of miniaturized camera systems for portable electronic devices and sensors demands not only the shrinking of the electronic and the opto-electronic but also of the optical components. A short focal length as well as a low number of optical components are the basic requirements to achieve that goal. However, the ongoing miniaturization of the image sensor size also causes a demand for a higher image resolution and light sensitivity. We demonstrate a miniaturized camera with approx. VGA resolution (700x550 pixels) that conceptionally follows an alternative approach of microoptical imaging systems. The so-called electronic cluster eye (eCley) captures different portions of the desired FOV within multiple, separated optical channels. These are subsequently joined together digitally to form a total image of the full FOV [see Fig. 8(a) and 8(b)]. The proposed segmentation enables a shorter focal length for a given FOV without scaling down the image sensor pixel size. Thus, a short total track length of only 1.4 mm has been realized which is about two times shorter than comparable single-aperture optics. On the other hand, the short focal length and the small field size per channel allowed for the application of simple optical components such as reflow microlenses with small sags. These have been fabricated with well-established microoptical fabrication techniques which #133960 - $15.00 USD

(C) 2010 OSA



exhibit sub-micron precision and are cost-efficient when it comes to high volume production due to wafer-level manufacturing. Alignment and assembly of the optical component wafers have been done on wafer-level. Additionally, it was shown that the distortion of the individual partial images could be corrected by means of image processing. This is a necessity for the image stitching algorithm and the creation of a final image that is free of distortion. Within future work, we are going to address the refinement of the image quality and resolution of the eCley which is diminished by limitations of the current algorithm for partial image stitching. Furthermore, we are going to extend the basic working principle of eCleys to achieve camera optics for one or two megapixels resolution with extremly short total track lengths. Acknowledgments We appreciate the funding by the German Federal Ministry of Education and Research (BMBF) for the project “Insect inspired imaging,” (FKZ: 01RB0705A) within the BIONA initiative. Furthermore, the authors like to express their gratitude to Sylke Kleinle, Antje Oelschläger, and Simone Thau who participated in the fabrication of the microoptics modules as well as Alexander Oberdörster for the real-time implementation of the fusion algorithm.

#133960 - $15.00 USD

(C) 2010 OSA

