REGULARIZED
ESTIMATION
OF OCCLUDED
Damon
L. Tull
DISPLACEMENT
Aggelos
VECTOR
FIELDS
K. Katsaggeios
Department
of Electrical and Computer Engineering Northwestern University Evanston, Illinois 60208-3118
[email protected]. edu, aggkfleecs.nwu.edu
ABSTRACT Occluded
regions
and
placement
vector
reconciled
to accurately
+A)z: ;; [@([vu(i, j)l) + W7qi.i)l)]
motion
field (DVF)
boundaries
introduce
discontinuities
estimate
image
flow.
dis-
that must be In this work,
the robust regularized estimation of the DVF is considered in the presence of these discontinuities. A robust convex estimation criterion is presented that preserves motion boundaries and allows for a globally A new class edge
of robust
preserving
gradient sequences
convex
and
of the DVF.
is introduced
an occlusion
as mechanism
due to occlusion.
estimate
measures
regularization
is proposed
continuities
optimal
for
weighted
for managing
Results
DVF
using synthetic
dis-
image
are presented.
where A is the regularization parameter, and lVv(i, j) I and IVU(Z, j)l represent the magnitude of the gradient of the vertical and horizontal components of the DVF, respectively. The addition of the (~) terms in Eq. (2) imposes a level of smoothness
on the motion
In traditional
(Tikhonov)
rj and ~ are chosen 5., unfortunately tion estimators ous features
(unweighed)
applications
including
cal imaging.
Discontinuities
used to estimate image scene. essential
image sequence
and filtering),
The effective
(depth)
Most traditional formulations izontal and vertical components U(Z, j), respectively ference (DFD), DFD(i,
j)
and/or
management estimation
(compres-
vision
convey information
the structure
for the accurate
processing
machine
and medithat can be
motion
of an
of discontinuities
is
of these features.
attempt to find the horof the DVF, V(Z, j) and
that minimize
the displaced
frame dif-
=
p(i,j) - p-’(i -
=
f:(i.m(i.o + f:(~jj)u(i~) + ff!
U(i, j), j -
v(i,j)) (1)
Dubois
measures
bet ween their robust
mulations proved
are obtained
rion to be minimized
authors
considered
iterative ited
performance
Although
the
and M RF forsignificantly
formulation
im-
the crite[1, 2] and in
non-convex approach
of ~ which
of the globally work,
~ (@ quadratic)
regulariz at ion algorithm.
of their
selection
8Q(u,
was attributed only
optimal
Rouchouze
et.
allowed
in an
The limfor
to the an ap-
DVF.
al. [5] and later Tull and
[6] suggested robust, convex@ functions which of edge preserving regularization. Selecting @ allows one to obtain a globally optiThe mechanism for edge preserving
regularization in this framework can be seen by taking the derivative of Eq. (2) with respect to U(Z, j) which gives,
criterion,
v, A)
Zirl(i, j)
NM
j=]
approach
in both formulations,
robust
minimization
C=l
and
[2] a ‘line process’
# and demonstrated
in the MRF
half-quadratic
and 4 to be convex mal DVF estimate.
~
is an
In Konrad
the robust formulation of [3] is non-convex, and have many local minima requiring costly minimization techniques that only approximate e a globally optimum solution. In [4] the
the DVF estimate accounts for differences in the frames of interest. The unconstrained global minimization of Eq. (1) is an ill-posed problem, requiring techniques such as regularization to obtain a unique (and meaningful) solution. A well-posed estimate of the DVF can be obtained from the
Q(u, r),A) = ~
for rj and
with a ‘line process’.
results
Katsaggelos are capable
DFD
and Katsaggelos
relationship
izontal and vertical) and temporal first derivatives of j~, respectively. The DFD is a non-linear measure of how well
of the regularized
mo-
discontinu-
of discontinuities
estimation.
non-convex
In recent
are the spatial
regularized
important
formulation. Black and Anandan in [3] first introduced the use of robust measures in Eq. (2). They proposed robust
proximation
of the pixel at location
and ~~, $$ and j$
both
in Section
was used to adapt their motion models in the presence of abrupt motion boundaries in a Markov random field (MRF)
(z, j)
is the intensity
Tikhonov
the management
(hor-
where jk (z, j),
As shown
of the DVF.
[1] and Brailean
non-convex in the kth frame
the DFD.
DVF estimation,
to be quadratic.
active area of research in motion
INTRODUCTION
The management, detection and estimation of discontinuous features in an image scene is vital to a wide variety of sion, interpolation
field that minimizes
regularized
tend to over-smooth
For this reason, 1.
(2)
i=l J=l
= #( DFD(i,
j))fj
+A
~
W(i,.i)
aprt(i,j)l ~u(z, ~,
,
e,j~e
rJ(DFD(i, j))
(3) ., where,
Copyright (C) 1996 IEEE.
@ is the support
over which
the derivative
All Rights Reserved.
is non-
trivial
and W(Z, j) is defined
as,
LJ(i,j) = d’(lv4i.ol) glvu(i,j)l ‘ where
+’ denotes
ferences
the first derivative
for spatiaJ derivatives,
the second
summation
(4)
of ~.
Using
the authors
Although
somewhat
functions inst ante,
for # successfully
some
adhere
previously
to these
first dif-
#GRN(Z)
=
log(cosh(~))
in [5] show that
#ENT(~)
=
(1~1+e-’)ln(lxl
proposed
conditions
for
(lo)
+ e-’).
becomes, The
(5) where Z(Z, j) is defined
from
authors
in [5] used 4GRN
j) = W(i, j – l)u(i,
+W(i, j)u(i,
+1, j)
j – 1) + W(i, j)u(i
j + 1)) +U(i
– l,j)u(i–
robust
and convex
ularization
motion
esti-
twice differentiable convex
measure
can be derived
for all z.
for edge
from
preserving
the error function
reg-
defined
as,
(6)
I,j),
for regularized
was introduced , an “entropic” class mation. In [6] #ENT of functional [8] to robust motion estimation. Both are
A new robust WU(i,.i)ti(i,
restrictive,
z
and W(Z, j) is defined
It can
erf(z)
as,
/ ..6
Wu(i, j) = W(i, j – 1) +Zu(i,j)
+W(i
be
LOU(Z,j)it(i,
weighted
seen
from
Eq.
(6)
sum of the motion
neighborhood
of (z, j).
not want motion
that
j)
is the
field at pixel locations
in the
To preserve
at pixels
(7)
– 1,.i)
across
edges in the DVF,
we do
the edge to influence
A closed form for this integral, tive distribution
function
does not exist.
Further
expressed
to the smoothness
this it should
the edge preserving
nature
In the following for edge preserving convex
measures
ularization.
penalty
at the center.
be clear that the structure
of@
Section
are suitable
3. describes
for edge preserving
a robust
iterative
regions
where the DVF is undefined
manage
occluded
gradient
is incorporated.
to synthetic
regions
sequences
in the DVF
es-
in Section
algorithm
is applied
motion
in Section
CLASS
OF
ROBUST
selection
of @ is non-trivial. gradient,
must
gument,
the spatial
removing the local
the influence of an edge pixel in the surround of computation of the smoothness in Eq. (6). In [7]
vanish,
several requirements for convex edge preserving measures are present ed. The first constraint is placed on the asymptotic behavior of ~’ in the form,
A sufficient
be convex
x-m
4’(Z)_ ~ —_ x
lim $+0
$$’(~) = (-J> —
().
z
for edge preserving
the minimum
selection
condition
From
this fact,
its second shown
regularization
(12) of
of Q4is convex
f(z)
for a function
f“(a)> O,vz6 Y?. Eq.
derivative
that
@IERF
(11)
shows
4TERF
is the normal also complies
(8) and (9) for convex the first integral
to
to be convex
distribution. with
robustness of CDFS
(13) for
It can be
the constraints
in
in this formulation.
as a family
of robust Due na-
ture of I#IERF. Figure 1 illustrates the robust structure of this proposed functional. In comparison to the quadratic, @IERF has a sknificanW srnfler P-h for large values of the argument while approximating the quadratic for smaller values. The first derivative (i$IERF achieves its asymptotic values faster than the previously proposed @ ENT and d GRN functions. The weighting functions (# ’(z) /z) in Figure l(r.)
and #GRN
(8)
are
considered in the context of anisotropic diffusion. In their analysis, an additional (weak) constraint was imposed on the second derivative of ~ to avoid smoothing across edges,
Copyright (C) 1996 IEEE.
The
to Vanish in a similar manner.
ratio of Eq. (9) for dIERF
for much smaller considered
1The
(9)
seem
The difference can be seen in Figure 2 were the derivative constraint of Eq. (9) is plotted against gument.
In [7], the conditions
be
is,
for @IERF ]im
to make
convex functional is of significant theoretic int crest. to space limitations we only consider the qualitative
For large values of its arthe # function
~–1 12 is included
zero at z = O. This
Eqs.
CONVEX
can only
of the er~ exists
+7r-1’2f3-’2 – rr-]’z.
x*erf(z)
and robust. ]
5.
FUNCTIONAL The
erf(z)dz
constant
Clearly, A NEW
2.
the cumuladistribution,
of the erf
The integral
this function
4. To
weighted
processing
r =
motion for oc-
an occlusion
The proposed of occluded
reg-
DVF
timation algorithm that successfully manages DVF boundaries. We extend this approach to account cluded
The
section, we discuss the conditions on ~ regularization and present a new class of that
41ERF(~)=
From
solution.
of the normal
and takes the form,
the
will determine
of the regularized
which represents
(CDF)
in terms of the erf.
cost of smoothness at the present location (i, j). That is, if the gradient of a pixel in the surround is large, it should not contribute
(11)
~e-’’dt.
=
—er.f (z)
values of the argument
second its ar-
is near w4@le
than for previously
~.
erj(z)
function
is
allowing for negative
odd
symmetric,
e~~(–z)
values of the argument
modification.
All Rights Reserved.
=
without
4. The
AN OCCLUSION WEIGHTED SMOOTHNESS CONSTRAINT
algorithm
of the previous
discontinuities
in the DVF.
section
However
effectively like most
manages motion
es-
timation algorithms, motion is assumed to be defined for all pixels in an image. In practice, this is not the case. Occluded
Figure
1.
su;es.
(r.)
(1.)
Robust
tions.
Quadratic
Robust
ted), IERF
quadratic and
and
quadratic
(solid),
Entropic
(dash),
such as uncovered
background
or new ar-
meafunc-
sion weighted
robust
weighting
regions
eas unrelated to motion are common in (video) image sequences. Such areas should not impact the DVF estimate at locations where the motion is defined. To manage the discontinuities due to occlusions using the edge preserving mechanism described in Section 1. we introduce an occlu-
LogCosh
gradient,
(dot-
(dot- da;h).
Ivzlo(i,’j)l’ = K(o(i,
j+
l))uz(i,
j)’
+K(o(i+
l,j))%(~,j
)’> (16)
where
the function
~(o(i, j))
the value of the occlusion
O(Z,j) 6 [0, 1], is
= [1, ~~~~],
field at pixel location
(z, j)
and
~ ~~~ is a large positive constant. If a neighborhood point is found to belong to an occluded region (where O(Z,j) tends to zero) K(O(Z, j)) tends to ~~~., removing its effect on the local smoothness calculation. We estimate the occlusion field from an initial
O(i, j)
Figure for
2. Robust
Quadratic
IERF
3.
and
quadratic
(solid),
weighting
Entropic(dash),
(~_ ~ _
2Qrag(i,
j)
)
s@(24er~(eedge(~) 2Wge(i,
(
field using,
d)
w4f+~f(~tTag(i
functions
Log f30sh (dotted),
of the motion
~)) j)
(17) )
(dot-dash).
A ROBUST
In [3], Black terms
ITERATIVE DVF ALGORITHM
and Anandan
bust functional
for both
IL allows
assumptions purposes
Letting 4= vertical
component
the DFD
that
are often
of the DVF
Using
the Horn
gives the vertical
Wg.(i.i)
optimal
For the
estimate
is obtained
of the
by setting
tensity
and Schunk
=
Aqi,
of occlusion
horizontal
component
approximation
To
is obtained
manage
xl(i,
(.f:(i, j).f$(i, J + ft(idf:(i d) Xh(i,.i) + f$(i,.i)z (15)
Equations
(14)
the occlusion 5.
on a uniform j) –
and (15)
are evaluated
the entire image until the change predetermined threshold.
for each pixel
in the DVF
of ~(.)
simple
defined
per-
measure
as the linear
(19)
= 1 + O(i, j) * (Kmax – 1).
over
is below
occlusion
j)l in Eq. (3).
a
Copyright (C) 1996 IEEE.
regions Replace
weighted
Vu(i,
j)
AND
a uniform
background.
is
replaced
with
u with v in Eq. (16) to obtain
the horizontal
RESULTS
Figure 3 illustrates
from,
?J(2,j) =
This
of
of the DVF,
of the DVF
pair.
in in-
function, K(o(i, j))
j) – (f:(i,.i).ft(i, j) + ft(i.o.f:(i.i)) AJAJ(i, j) + R(i, j)’ motion
is the argument
the deviation
and the temporal
the
(14) similarly
of motion
sist ence of edges in the image
Vuo(i, ‘u(z’,j)
- Ivfk-’(i - ZL(i, j),j - V(i,j))l.
ct,~g and ~.dg. measure
in the direction
(3) to zero and solving
component
Ivfk(i,j)l
=
The quantities
required
of noise.
we select
of Q(M, v, A) in Equation
for U(Z, j).
note that US-
in the underlying
in the presence
The globally
(18)
~)
~ quadratic as in [5, 6]. r#lERFmakes the regularized DVF criterion of
(2) convex.
derivative
They
deviations
constraint
the use of ro-
(~) and motion
formulation.
flow estimation of this work
Equation
first considered
for slight
of motion
for optical
ESTIMATION
the smoothness
in their regularized
ing robust
The
=
estimate
motion
component.
DISCUSSION
object
translating
In this image
pair,
to the right the uncov-
ered background is identical to the (previous) surround, in a sense, masking the occlusion effect. In theory, the DFD can be minimized to zero for this image pair. Figure 4 illustrates the DVF estimates obtained using ~(z) = X2 (left) and O(X) = q51ERF (right)
the over smooth This is expected
without
estimate because
occlusion
management.
Note
obtained in the case of @(z) = Z2. the quadratic equally weights the
All Rights Reserved.
smooth areas and edges in the estimate as shown in Figure 1. The robust convex selection of @ = CJ51ERF allows for
segmentation
DVF edges while smoothing the motion field by discounting the penalty for DVF edges. Although the robust estimate results in a DVF more useful for higher order processing (segmentation, region coding), in a strict sense, both approaches make errors in their estimates in that motion is
sion region detect ion [9, 10] is also under investigation.
imposed
in areas where motion
was experiment
ally chosen
is undefined.
to be equal
In all tests,
is expected
information
by further
into the robust
embedding
formulation.
occlusion
Enhanced
occlu-
-. —. —: —. --.__=
A
1.1.
Figure 5 shows the same object moving with anomalous background exposed due to motion. Clearly, motion for the bright region exposed in the second frame of the image pair is undefined.
This
region
ance is irreconcilable
is occluded
because
its appear-
frame.
The robust
using the previous
estimate of the motion field with and without weighted gradient is shown in Figure 6.
the occlusion
Robust estimate with exposed backFigure 6. ground without occlusion management (L), estimate with occlusion management (r.). REFERENCES [1] J. Konrad
and E. Dubois.
tion vector
fields”.
“Bayesian
IEEE
Trans.
estimation
on PAMI,
of mo-
14(9):910-
927, September 1992. [2] J.C. Brailean
Figure
3.
right).
Note:
Horizontal
motion
Uniform
(block
moving
left
backgroundmasks occlusion
to
effects.
[3]
and A.K.
Katsaggelos.
stationary
map displacement
gorithm”.
IEEE
429,
1995.
April
Trans. on Image
M.J Black and P. Anandan. of motion
Vision,
Processing,
“A model
In Proc.
over time”.
“A recursive
Int.
al-
4(4):416-
for the detection
Conf.
on Computer
pages 33-37, Osaka, Japan, December 1990.
[4] L. Blanc-F6raud, M. Barlaud, and T. Gaidon. tion
non-
vector field estimation
estimation
resolution
involving
scheme”.
discontinuities Optical Engineeringj
“Mo-
in a multi7(32):1475-
1482, June 1993.
[5] B. Rouchouze, Figure 4. (1.) Horn and Schunk estimate @ and ~ quadratic. (r.) Robust regularized estimate ~ = @E~F, @ quadratic. Uniform object, uniform background.
laud.
P. Mathieu,
“Motion
estimation
T. Gaidon, based
and
on markov
M. Barrandom
fields”. In International Conference on Image Processing, volume 2, pages 270–274, November 1994.
[6] D.L. Tull and A.K. estimation
Katsaggelos. using robust entropic
ternational
Conference
pages 212-215,
on Image
October
“Regularized functional”. Processing,
motion In involume
3,
1995.
[7] L. Blanc-F6raud, M.
P. Charbonnier, G. Aubert, and “Non-linear image processing: model-
Barlaud.
ing and fast algorithm tection”.
In IEEE
age Processing,
Figure 5. Horizontal motion (block moving left to right) exposing anomalous background. CONCLUSION
6. Occluded defined,
regions, should
where the motion
not impact
is (instantaneously)
the motion
estimate
un-
where mo-
tion is well-defined. Most motion estimation algorithms, assumes that the DVF exists everywhere in a scene. This is assumption
often violated
for incorporatin~ bust
convex
DVF
new class of robust liminary
results
in practice.
In this work, a method
knowledge
of occluded
estimation
criterion
convex
measures
are promising
regions
into
was presented was derived.
but limited.
a ro-
and a
Our pre-
Improved
DVF
Copyright (C) 1996 IEEE.
D. C., USA,
for regularization International
volume
October
with edge de-
Conference
1, pages 474–477,
on Im-
Washington,
1995.
[8] M.E. Zervakis,
A.K. Katsaggelos, and T.K. Kwon. “A class of robust functional for the enhancement of imIEEE Trans. on Image Processing, 3(14):752ages”. 773, June 1995.
[9] R. Depommier with detection
and
E. Dubois.
of occlusion
areas”.
“Motion estimation In Proceedings of the
Conference on Acoustics, Speech IEEE International and Signal Processing, pages 269–272, March 1992.
[10] S.L. Iu. “Robust
estimation
continuity y and occlusion
of motion
vector
with dis-
using out lier reject ion “. Jozw-
and Image nal of Visual Communication tion, 6(2):132-141, June 1995.
All Rights Reserved.
Representa-
REGULARIZED
ESTIMATION
PLACEMENT Damon
VECTOR
L. Tull
Department Northwestern Evanston,
and Aggelos
of Electrical
OCCLUDED
DIS-
K. Katsaggelos
and Computer
Engineering
University
Illinois
60208-3118
[email protected]. Occluded
OF
FIELDS
edu,
[email protected]
regions
and
motion
boundaries
introduce
dis-
placement vector field (DVF) discontinuities that must be reconciled to accurately estimate image flow. In this work, the robust regularized estimation of the DVF is considered in the presence timation
of these discontinuities.
criterion
aries andallows A new class edge
for a globally of robust
preserving
gradient
is presented convex
regularization
is proposed
optimal
convex
motion
estimate
measures
of the DVF.
for managing
Results
for
weighted DVFdis-
using synthetic
Copyright (C) 1996 IEEE.
es-
bound-
is introduced
and an occlusion
asmechanism
continuities due to occlusion. sequences are presented.
A robust
that preserves
image
All Rights Reserved.