Efficient neural models for visual attention

Efficient neural models for visual attention Sylvain Chevallier, Nicolas Cuperlier and Philippe Gaussier ETIS - Neurocybernetic team Univ. Cergy-Pontoise – ENSEA – CNRS Cergy, France [email protected]

September, 22th. 2010

Framework

Outline

1

Framework Visual attention Neural models

2

Models and implementation Attentional architecture Implementations

3

Experimental results

S. Chevallier (ETIS)

Efficient neural models


2 / 20

Framework

Visual attention

Change blindness




3 / 20

Framework

Visual attention

Change blindness




3 / 20

Framework

Visual attention

Change blindness




3 / 20

Framework

Visual attention

Bio-inspired attentional vision systems Attentional spotlight metaphor Reduce the search space [Tsotsos, 90] Attentional architecture Feature extraction Combination on saliency map Focus selection through Winner-Take-All

[Itti & Koch, 98]

Applications Driver assistance [Michalke, 08] Retinal prostheses [Parikh, 10] Robotics [Frintrop, 06] S. Chevallier (ETIS)



4 / 20

Framework

Neural models

Bio-inspired information coding Neurons exchange information through spikes

Spikes have little variations in amplitude and duration Spikes are fully characterized by their emission dates Level of description for neural models: Neuron level Temporal coding, precise spike timing Population level Rate coding, mean firing rate S. Chevallier (ETIS)



5 / 20

Framework

Neural models

Neural models

Spiking Neuron Network Network of [1, . . . , i, . . . , N] spiking neurons: ( P P (s) dVi j∈Pre wij s∈Trainj δ(t − tj ) + I(t), if Vi < ϑ dt = −λi Vi (t) + else trigger a spike and Vi ← Vreset Frequency-based Neural Network Continuum neural field τ

∂u (x, t) = −u(x, t) + ∂t


Z

w(x − x0 )f [u(x0 , t)]dx0 + I(x, t) + h



6 / 20

Framework

Neural models

Goal of this paper

Question What is the most suited neural coding scheme for an efficient bio-inspired attentional architecture ?

Compare SNN and FNN Complexity analysis Quality of results Simple artificial images Natural images




7 / 20

Models and implementation

Outline

1


2


3





8 / 20


Attentional architecture

Preattentive visual architecture IOR

Input image

Low spatial frequencies

WTA

Saliency Input maps

High spatial frequencies

Multi-scale Features

FNN needs WTA to sort saliencies

Contrast of luminance (DOG) Orientations (Gabor) Color opponencies (DOG) S. Chevallier (ETIS)


SNN is an anytime process September, 22th. 2010

9 / 20


Implementations

SNN implementation

DOG filter

details

Neural filter S. Chevallier (ETIS)



10 / 20


Implementations

Complexity analysis FNN Filtering cost: for f features, s spatial scales, filters of size M and N input image pixels WTA cost: O(N) with ARGMAX O(f × s × M × N) SNN Hybrid synchronous simulator, with time step ∆t Total cost = Spike propagation cost + neuron update cost cp × F × M × N + cu ×

A ∆t

F is mean firing rate, A is number of active neurons. cu is 10 FLOP. S. Chevallier (ETIS)



11 / 20


Implementations

Complexity analysis SNN computational cost depends on emitted spikes Is the number of spikes constant for processing different images ?

CPU cycles (106 )

2.5

1 patch 10 patchs 50 patchs 100 patchs

2 1.5 1 0.5 0

0

10

20 30 40 Simulated time (t)

50

60

For SNN, computational cost depends on the input image Rich images (w.r.t chosen filters) induce large number of spikes S. Chevallier (ETIS)



12 / 20


Outline

1


2


3





13 / 20


Comparison on artificial images Pop-out artificial images

FNN Circle shows most salient region, winner of WTA (FNN) SNN Dots indicate the most salient pixels (SNN) Same salient items are found for FNN and SNN (20 images) S. Chevallier (ETIS)



14 / 20


Natural images 19 webcam images of 160x120 pixels

Salient regions might not be extracted in the same order Measured computational cost (as CPU cycle): Constant for FNN SNN can find salient regions before FNN (1/4 of the images) S. Chevallier (ETIS)



15 / 20


Conclusion and perspective Comparison of two neural models for an attentional system Frequency-based Neural Network: have a constant and lower computational cost, needs a WTA to sort saliencies Spiking Neuron Network: have a variable computational cost have anytime capabilities Perspective Formal analysis of spiking neuron processing Learning capability of neural network Attentional bias modulating salient regions Long term adaptation of input signal (slow variation of illumination) S. Chevallier (ETIS)



16 / 20


Annex




17 / 20


Input maps

dVi dt

= −λi Vi (t) + KLi , if Vi < ϑ else trigger a spike and Vi ← Vreset

with Li the considered pixel value Φi =

1 λϑ ˆti = − ln 1 − i λi KLi

λi

= −

ln 1 −

back

≈


1 ˆti


λi ϑ KLi

K Li ϑ


18 / 20


Input maps




18 / 20


Integration maps

PPj = −λj Vj (t) + i=1 wij Si (t), if Vj < ϑ else trigger a spike and Vj ← Vreset dVj dt

Si (t) =

Ni X

δ(t − tif )

f =1

back




19 / 20


Integration maps

PPj = −λj Vj (t) + i=1 wij Si (t), if Vj < ϑ else trigger a spike and Vj ← Vreset dVj dt

Vj (t) =

Pj X

wij

Ni X

e−λj (t−fˆti ) H(t, f ˆti )

f =1

i=1

Vj (Tj ) ≈

Pj X i=1

wij

1 − e−QNi /Li 1 − e−Q/Li

with Q =

λj ϑ K

back




19 / 20


Frequency coding

P1 P2 P3 P4

V

ϑ S t ISI


6 ms

4 ms

5 ms


5 ms

4 ms


20 / 20