Image processing in fluid mechanics by CMOS

Home

Program

Go to Previews View

Proceedings of PSFVIP-4 June 3-5, 2003, Chamonix, France. F4042

Image processing in fluid mechanics by CMOS image sensor

1*

1

2

3

Lionel Lelong , Guy Motyl , Julien Dubois , Alain Aubert and Gérard Jacquet

1

1

Laboratoire Traitement du Signal et Instrumentation (TSI), UMR CNRS 5516, Saint-Etienne, France. Ecole Polytechnique Fédérale de Lausanne (EPFL), Lausanne, Suisse. 3 Ecole Supérieure Chimie Physique Electronique de Lyon (CPE), Lyon, France. 2

KEYWORDS: Main subject(s): Smart sensor, Fluid: Hydrodynamics, Visualization method(s): Particle Image Velocimetry, Other keywords: Real Time, Binary Correlation.

ABSTRACT : This paper presents a dedicated architecture which is devoted to measurements in PIV by real time image processing. It is based on CMOS image sensor and FPGA circuits. It allows to realise small, non expensive and flexible acquisition and processing systems. The system architecture is presented. Performances in term of speed and precision are given. Sub-pixel performances can be attained with video rate more than 100 images per second with reduced acquisition areas. The flexibility of the system allows to realise a compromise between speed and resolution without materiel change.

1. Introduction Measurement and control by image processing imply real time operations on high data flow. Many algorithms in image processing can not be achieved in real time with a PC. Nowadays, the rising performances of the high density electronic devices enable to realise more and more hardware architectures in order to reach real time computation for low cost. These architectures must be developed with concept of Architecture Algorithm Adequacy (AAA) [1]. Real time computation in PIV techniques allows to extract motion vector fields at the video rate (25 images per second or more). This is only possible with a dedicated electronic architecture. More and more applications as automatic system control or real time visualisation can take profit of this computation rate. A specific system, called Round-About, was built in the laboratory to take into account the different characteristics of real time applications [2]. For that, it has a large flexibility from the point of view of the number of operating units but also of their nature: DSP (Digital Signal Processor) or FPGA (Field Programmable Gate Array). This system is dedicated to image processing, the acquisition part is realised by an application specific board included in the system, the results are sent to a PC by a normalised link. Generally, the images come from a standard CCD camera and are stored in FIFO memories. The use of CCD cameras limits the flexibility of the system because the previous image must be entirely read before the next image is accessible. Moreover, management of the acquisition *

Corresponding author: Lionel LELONG, Laboratoire TSI,UMR CNRS 5516 / Université Jean Monnet, 21 Rue du Docteur Paul Michelon, 42023 Saint-Etienne Cedex 2, France – email: [email protected] Tel : 00 33 (0)4 77 48 51 71 – Fax : 00 33 (0)4 77 48 51 20

1

Copyright 2003 by PSFVIP-4

Home

Program

Go to Previews View


parameters needs a complex synchronisation system [3] and non standard video signals come from high speed CCD cameras. New CMOS image sensors (CIS) offer the possibility of random addressing of small areas on the pixel array and allow fast processing of region of interest. In motion measurement application, the use of CIS allows to realise new systems with high performance in image processing. This paper presents a dedicated architecture which is devoted to measurements in PIV by real time image processing. Based on CMOS image sensor and FPGA circuits, we propose small, non expensive and flexible acquisition and processing systems.

2. Previous Experimental Set up Figure 1 shows a typical PIV system used for the estimation of the flow motion in a plane while illuminated, by a laser sheet. A CCD camera records the position of the particles seeding the flow. The aperture time of the camera and other timing parameters are controlled by a synchronisation system [3]. After digitisation, the images are recorded in a PC memory. Then, the determination of the motion vector field is made by an image processing software on a workstation [4]. Laser Particles Laser sheet

Blower

Jet of smoke +

Particles

CCD Camera PC

Synchronisation system

Fig.1: The measurement system in PIV. To reach real time performances with this system, it is necessary to insert a specific system between the CCD camera and the PC. This system receives the video flow and realises the real time computation, only results are sent to the PC and displayed (figure 2). CCD Camera

video

synchro

Acquisition and processing system

result command

Synchronisation system

PC Fig.2: Diagram of real time system. Specific systems have been developed. One of them, based on a Round-About architecture [5], allows different kind of applications because of its modular architecture (figure 3).

2


Home

Program

Go to Previews View


Cameras Data dispatching module

PM Data flow PM

Control module

PM PC PM

PM : Processing Module Command and result flow

Fig.3: Diagram of Round-About architecture. Real time processing on image rate of 25 images per second can be attained with 8 DSP TMS320C60 processor with a standard cross-correlation PIV algorithm. Results for 100 images per second rate is possible with simplified algorithms implemented on PLD (Programmable Logic Device) or FPGA.

3. System with CMOS Image Sensor 3.1 CMOS Image Sensor The use of CMOS technology as image sensor is recent and the level of integration follows the evolution of traditional CMOS fabrication processes. This evolution implies a significant reduction of minimum size of the pixels (photodiode and read-out circuit). Even if CCD technology has various advantages like higher resolution (~ 16 Mega pixels), reduced pixel size (lower at 10 µm) and fill factor up to 100%, the disadvantage of CCD image sensors for realisation of real time system is the need to transfer the full image to image processing unit before to have access to the following image. Moreover, the implementation of the CIS is simpler, the signal levels are compatible with the digital devices and the sensor has a simplified numerical interface as an electronic memory circuit. Finally, the possibility of integration of all the image processing within the CMOS image sensor on the same chip (Smart Sensor) is of great interest in future industrial applications. CIS is organised like memories (figure 4): each pixel of the matrix is directly addressable according to lines and columns selection by access decoders and switches, the signal being conveyed on vertical data buses. Current CIS are called Active Pixel Sensor (APS) because, inside the pixel, read-out circuit has an amplifier which provides the signal of the detector at a reduced impedance to the vertical data bus. Charges-tension conversion is thus realised inside a pixel. Before to go out the sensor, the pixel signal is digitized by an on-chip Analogic Digital Converter. The architecture of CIS offers significant advantages: a parallel architecture at level of columns, a rather low power consumption, no blooming effect and random pixel access. This random pixel access enables various possibilities like windowing, application of an image processing algorithm on each parts of the image, or tracking of an object in a region of interest. Windowing allows to cut out the image in small windows of identical or variable size directly on the matrix of pixels, or to limit to a region of interest which is a significant gain for image processing. With windowing, very high rate level can be attained on small part of images and therefore, it is possible to follow local physical event like turbulence.

3


Home

Program

Go to Previews View


Fig.4: architecture of the image sensor core with pixel selection principle [6].

3.2 CMOS acquisition and processing system The realisation of the global system needs main functions: acquisition and processing. Acquisition functions are based on logical signals as synchronisation signal and multilevel signals to realise control of gain or offset. In CIS, these controls are digital which it results a simplification of their implementation in electronic architectures. Processing functions imply arithmetic computation. An implementation on a programmable logical device as an FPGA makes possible to realise all these functions on the same circuit at a reduced cost. Due to its capabilities of integration of arithmetic and logical functions, FPGA brings the possibility to develop structures dedicated to the real time image processing. Moreover, the use of standardised programming language as VHDL (Very high speed integrated circuit Hardware Description Language) provides a great flexibility. Therefore, we obtain a modular architecture from the point of view of associated electronic devices, but also from the point of view of image processing algorithms. To reach real time performances with this system, a simple architecture can be used (figure 5):

CIS

On-chip control

10 Commands

Image sensor core

10 bit ADC

Acquisition Unit

Static RAM

FPGA

Programmable Interconnection

Image Processing

Data [0..9]

Interface with PC

Fig.5: Diagram of our architecture.

4


Home

Program

Go to Previews View


4. Adaptation of Particle Image Velocimetry technique 4-1 Grey level Cross-correlation In standard PIV systems, the motion vector field is estimated by a technique of crosscorrelation. Cross-correlation is carried out on study windows of variable sizes: 16x16, 32x32 or 64x64 pixels. The size of the window for a fixed image size gives the number of vectors. Each motion vector is obtained by cross-correlation of two extracted study windows of the same zone in two consecutive images. The algorithm extracts a pattern from the study window of the first image, and then determines the displacement of this pattern on study window of the second image that yields the best correspondence. Practically, it finds the most probable position of this pattern in the second study window by seeking the maximum of the function of correlation F:

F (i, j ) = ∑ x

∑ s1(x, y )× s 2(x − i, y − i )

(1)

y

s1 and s2 respectively represent the grey levels of the motif extracted from the first study window and the second study window. The localization of the maximum of correlation (peak) in correlation image determines the displacement between the two snapshots. This operation is repeated on all study windows of the image to obtain the motion vectors field. As the direct correlation has a high computation cost, the result is generally obtained by Fast Fourier Transforms techniques. An implementation of this method on a multiprocessors specific system [7] reaches real time performance. However this kind of implementation needs a complex hardware and a hard design process. Typical images are presented on Fig.6. Im age 1 with patterns 16° 16

Im age 2 with windows 32° 32

Extraction of a pattern and the corresponding window Result Im age Result window

Direct cross-correlation: Successive displacem ents of the pattern in study window

Fig.6: Extraction of a motion vector field.

5


Home

Program

Go to Previews View


Grey level real time correlation on 512x512 pixels images at a frequency of 25 images per second or more needs specific architectures based on several powerful processors working in parallel like TMS320C6x[7]. The results are the same than in a workstation computation but to reach real time the system is more complex and expensive. Simplified correlation can be realised with almost equivalent results if the computing window is not too small (16x16 or more). Then, an adaptation of the traditional method of correlation on grey level images is possible with binary images

4-2 Binary correlation It is necessary to initially transform the grey level images to binary images. The threshold choice is function of the median value of grey levels of the image. It allows to take into account the particles density of the images. In the case of a non homogeneous illumination, the choice of the threshold can be separately made for each study window. The operation of cross-correlation between two study windows then carries out by the research of the binary motif s1 in the binary study window s2 (figure 7). A direct comparison between the pixels is then carried out. For each position of the motif s1 in s2, the identical pixels are detected and counted. The maximum of final image corresponds to the most probable position of the motif in s2, and allows to directly obtain displacement between two acquisitions. The comparison between the pixels is carried out starting from not-or-exclusive logical operator (XNOR) and the count of identical pixels by simple adders. The algorithm (1) is modified by replacing the multiplication by logical operation XNOR:

F (i, j ) = ∑ x

∑ s1(x, y )

XNOR s 2( x − i, y − i )

(2)

y

This modification allows an implementation on programmable logical circuits. An acquisition and binary processing system can thus be inserted between the camera and the PC. Only the motion vectors are then sent to the PC. An architecture based on this principle is under development at the laboratory.

Pattern s1

Search for the pattern in the second image

Image s2

Fig.7: Cross-correlation.

5. Evaluation of the precision The precision of operation on binary images is evaluated with synthesized images [4]. It enables us to generate couples of images with a given sub-pixel displacement. The grey levels of each generated particle follow a bi-dimensional Gaussian law with a standard deviation of 1.5 pixels. Sub-pixel displacement estimation is made by a parabolic interpolation of the peak of correlation. Measurements are carried out starting from pairs of images having a given displacement and for particles densities varied of weak to strong. Displacements in X and Y were selected according to the errors related to the operator of interpolation. Displacement in X (of 3.3 pixels) corresponds to a maximum of error and in Y (of 2.5 pixels) to a minimum. A comparison between the various methods was carried out (Tab.I).

6


Home

Program

D Db XNOR

Go to Previews View


number of particles 13107 26214 52428 65533 X =3.260± 0.030 X =3.248± 0.029 X =3.232± 0.030 X =3.230± 0.030 Y =2.497± 0.021 Y =2.500± 0.023 Y =2.500± 0.029 Y =2.501± 0.028 X=3.214± 0.106 X=3.218± 0.080 X =3.217± 0.085 X =3.211± 0.084 Y=2.489± 0.131 Y=2.495± 0.124 Y =2.497± 0.115 Y =2.502± 0.111 X =3.216± 0.108 X =3.219± 0.082 X =3.218± 0.083 X =3.211± 0.083 Y =2.490± 0.131 Y =2.495± 0.123 Y =2.496± 0.114 Y =2.502± 0.112 D: Standard correlation on images to 256 grey levels Db: Standard correlation on images with binary images XNOR: Correlation with XNOR on binary images

Tab.I: Average and standard deviation on 256 motion vectors of window of size 32x32 (displacement X = 3.3 and Y = 2.5). In order to evaluate the intrinsic loss of precision due to the binarisation of the images, a comparison between the implementation of the direct method of cross-correlation on images on grey levels (line D) and images binary (line Db) was carried out. The results on a window 32x32 show that the maximum loss of precision is 0.05 pixels. The standard deviation of measurement is largely increased for the binary images (multiplied by a factor 4). It is interesting to note that there is no difference between the standard correlation and the correlation by operator XNOR on binary images. An accuracy of 0.1 pixel is easily reached while working on binary images in spite of the loss of information. The use of a larger study window (64x64) can make possible to attenuate the loss of precision related to binarisation (Tab.II). The precision is reduced in the order of 0.03 pixels, the standard deviation is decreased by the same factor for all the methods.

D Db XNOR

13107 X =3.271± 0.011 Y =2.500± 0.009 X=3.241± 0.046 Y=2.509± 0.067 X =3.240± 0.045 Y =2.509± 0.066

number of particles 26214 52428 X =3.258± 0.012 X =3.244± 0.016 Y =2.500± 0.012 Y =2.500± 0.015 X=3.224± 0.037 X =3.217± 0.046 Y=2.498± 0.080 Y =2.499± 0.069 X =3.224± 0.038 X =3.218± 0.046 Y =2.499± 0.078 Y =2.500± 0.067

65533 X =3.239± 0.015 Y =2.503± 0.013 X =3.223± 0.042 Y =2.497± 0.052 X =3.225± 0.042 Y =2.497± 0.054

Tab.II: Average and standard deviation on 64 motion vectors of window of size 64x64 (displacement X = 3.3 and Y = 2.5). The precision obtained on windows 32x32 or 64x64 is sufficient for many applications.

6. Implementation Based on the concept of smart sensor, our specific architecture associates an image acquisition unit with a CIS and a computing unit based on a FPGA. The image acquisition unit is a CMOS image sensor from the society FillFactory: IBIS4[6]. The IBIS4 is a digital CMOS active pixel image sensor with SXGA format. Their main features are: a high resolution of 1280x1024 pixels, a good fill factor of 60%, and a lower pixel size of 7 µm. The chip is composed of 3 modules: an image sensor core, a programmable gain output amplifier, and an on-chip 10 bit Anolgic Digital Converter. The frame rate is about 7 full frames per second. For an array of 512x512 pixels, a quarter of the full area, the image frequency reaches frequency of 25 images per second. The proposed system is a reduced system constituted with only one FPGA directly connected to the CMOS image sensor. This system offers significant advantages: all signals between CIS and FPGA are digital, and the architecture is based on a modular concept (all electronic devices are replaceable by more powerful devices).

7


Home

Program

Go to Previews View


Computing unit is based on a FPGA from the society ALTERA: APEX20KE [8]. This device combines the strengths of Look Up Tables (LUT) and product-term-based devices with an enhanced memory structure. LUT-based logic provides optimised performance and efficiency for data-path, registerintensive, mathematical, or digital signal processing designs. Product-term-based logic is optimised for complex combinatorial paths, such as complex state machines. LUT and product-term-based logic combined with memory functions make the APEX20KE device architecture suited for system-on-aprogrammable-chip designs. APEX20KE device is composed of 8320 logic blocks (typical 200000 logic gates), and 104 kilo-octets of RAM (Random Access Memory). Moreover, the user can connect 376 different signals to it. The structure of a FPGA consists of a logic network connected by programmable interconnections and surrounded by input/output blocks. A previous study has shown that the complete treatment of a pattern and a study window is carried out by a computing heart requiring 8500 logical gates. A computing heart realises a binary correlation on one window with a pattern size of 16x16 and a window size of 32x32. At a clock frequency of 12MHz, one computing heart makes possible to obtain a processing frequency equal to 13.5 images (512x512) per second. The reduced number of resources necessary for the implantation of a heart lets us consider the possibility of including several hearts in the same FPGA. Each heart works then simultaneously on distinct pattern/window couples. By using an APEX20KE FPGA, a maximum of 8 computation hearts can be integrated. This shows that an entire real time system at 100 images per second can be implemented in only one FPGA.

7. Conclusion and prospects Evolution of programmable electronic circuits now allows to realise an entire real time image processing system within only one circuit. The definition of regions of interest, made possible by the use of CIS, allows to only focus computing resources on the phenomenon to be studied. We will studied the aspect of tracking of object or of region of interest in the observation field of sensor. This leads to the definition of new concepts in the acquisition and processing of image data, particularly in high speed applications as PIV. Low cost industrial applications are then possible like real time flow parameter control and real time visualisation of dynamic flow experiments. References [1] R. Bourguiba, D. Demigny, M. Mamoud Karabernou, and L. Kessal, “Designing a new architecture for real time image analysis with dynamically configurable FPGAs”, Proc. IEEE -IMACS Computationnal Engineering in Systems Applications, Nabeul Hammamet, Tunisia, May 1998. [2] Julien Dubois, “De l’intégration d’algorithmes de traitement d’images pour la mesure temps réel du mouvement vers la définition d’une architecture générique”, PhD, France, December 2001. [3] H. Zara, V.Fischer, J.Jay, R.Fouquet, E.Tafazolli, G.Jacquet, “High speed video camera applied to flow study,” Meas. Sci. Technol, vol.9, 1998, pp 1522-1530. [4] “Wima: image processing sofware,” Laboratory TSI, Saint-Etienne, http://www.univ-st-etienne.fr/tsi/Wima. [5] J. Dubois, G. Jacquet, G. Motyl, F. Celle, and V. Fischer, “« Round-About »: une architecture de traitement d’images pour la mesure en temps réel de paramètres physiques“, Colloque GRETSI’01 (CD), Toulouse, France, September 2001. [6] http://www.fillfactory.com, extracted image of datasheet: http://www.fillfactory.com/htm/products/datasheet/ibis4.pdf [7] J. Dubois, G. Jacquet, G. Motyl, V. Fischer, and R. Fouquet, "System for real time motion measurement", The Third European DSP Education & Research Conference, Paris, Sept. 2000. [8] http://www.altera.com .

8
