The proposal is a new implementation of the Graduate Non-Convexity (GNC) ... The GNC was originally developed by (Blake and Zisserman, 1987) for a single ...
MULTIGRID INTERPOLATION AND DISCONTINUITY DETECTION IN HIERARCHICAL SURFACE RECONSTRUCTION Raid Al-Tahir Department of Surveying and Land Information The University of the West Indies Trinidad and Tobago Commission III
KEY WORDS: Surface Reconstruction, Surface Interpolation, Discontinuity Detection.
ABSTRACT Working within the framework of hierarchical surface reconstruction first introduced by (Schenk et al., 1991), this research focuses on the subsystem of surface interpolation and defines the tasks for interpolation and analysis of a surface. The main objective of this research is to develop an algorithm that produces a dense representation of a surface from a sparse set of observations and, at the same time, facilitates preliminary labeling of discontinuities in the surface. Additionally, this study establishes the instrument and the mechanism for the communication between the levels of the image pyramid as well as the subsystems of the hierarchical surface reconstruction paradigm. The suggested approach is the simultaneous interpolation of the surface and detection of its discontinuities by adopting the weak membrane model. The proposal is a new implementation of the Graduate Non-Convexity (GNC) algorithm based on multigrid techniques. The GNC was originally developed by (Blake and Zisserman, 1987) for a single grid, for applications in computer vision. As it was initially proposed, GNC depends a great deal on the optimal setting of the control parameters. In this work, a specific procedure has been developed that determines the control parameters automatically and less subjectively.
1 INTRODUCTION In digital photogrammetry, surface reconstruction is the frame that enclaves all the tasks for modeling the portion of the real world that has been photographed. Automatic surface reconstruction entails two major problems: determining conjugate points or features (matching) and densifying the matched points in object space (interpolation). The two tasks are usually performed sequentially in a hierarchical approach, without interacting with one another except for providing an approximations to the next level of hierarchy. In order to improve the success rate and the reliability of automated surface reconstruction the results from densifying and analyzing the surface in any level must be considered by the matching on subsequent levels. The tasks of interpolation and surface analysis may give clues about surface discontinuities and occlusions - a vital feedback for the matching process. The conceptual framework developed in (Schenk et al., 1990; Schenk and Toth, 1991) for surface reconstruction possesses several appealing features. The most distinguished features, as related to this study, are its modularity and hierarchy. For these reasons, this study adopts this framework to define the type of input, output and the expected tasks for surface interpolation subsystem. This subsystem is of particular interest for reconstructing surfaces from large-scale aerial scenes of urban areas. Because of man-made features and buildings, aerial imagery of urban areas is usually associated with a higher percentage of occlusion, repeated patterns or similar features, and many surface discontinuities. All these problems affect the success rate of any matching algorithm. Besides, there is no straightforward answer to the selection of a specific surface
interpolation method characteristics.
that
would
accommodate
such
The objective for this study is then to develop an algorithm that produces a dense representation for a surface from a sparse set of observations and, at the same time, facilitates preliminary labeling of discontinuities in the surface. Moreover, this study caters the base for the flow of information between different modules in the hierarchical surface reconstruction paradigm.
2 SURFACE INTERPOLATION In digital photogrammetry, surface reconstruction is the frame that incorporates all the tasks for modeling the portion of the real world that has been photographed. Automatic surface reconstruction entails two major problems: determining conjugate points or features (matching) and densifying the matched points in object space (interpolation). While there are several approaches for reconstructing surfaces from digital images, the operation and success of all of these approaches depend upon having a dense representation for the surface. A scheme for surface interpolation, hence, is required for furnishing the needed representation, and facilitating further analysis of the surface. It is clear that the objectives of surface interpolation are • Constructing as realistic a surface representation as possible. • Preserving essential surface characteristics implied by the observations. A violation to this may occur when a smooth surface is interpolated over breaklines.
•
Refraining from introducing new characteristics to the surface, for example, creating new maxima or minima in the surface. Utilizing a priori information on discontinuities. If information is available, or becomes available during the process, then there must be a mechanism to incorporate them in the solution.
•
The number of methods for interpolating data over a grid is large if one considers all different variations and solution methods. In previous studies by the author, (Al-Tahir and Schenk, 1992; Al-Tahir, 1996), several methods were reviewed, tested and evaluated according to the criteria mentioned previously. The conclusion was that interpolating a surface by thin plate (or membrane) splines within a relaxation scheme fulfills the criteria for a good surface interpolator, and provides explicit information for surface analysis as well. The thin plate splines are obtained by solving the small deflection equation of an infinite plate that deforms under a bending moment due to point loads only. Adopting the physical analogy, depth data are represented by a set of vertical pins scattered within the region; the height of an individual pin is related to the elevation of the point. Fitting a surface is then analogous to constraining a thin plate to pass over the tips of the pins (Terzopoulos, 1983). The other type of splines is represented by the following differential equation ∂ s2
∂ s2
∫∫ ∂ 2 x + ∂ 2 y dx dy ,
(1)
which corresponds to the small deflection energy of a membrane. The physical forces in the membrane are due primarily to its surface tension. Thus, it generates minimal area surfaces that are continuous, but need not have continuous first partial derivatives. The membrane is found, (Al-Tahir and Schenk, 1992; Al-Tahir, 1996), to be more suitable than the thin plate to the problem of surface interpolation since it does not overshoot depth constraints. Since the goal for this step is not the smoothing of the data but filling the gaps between them, the membrane is formulated as Laplace equation with Dirichlet boundary conditions. The boundary condition is defined as si,j=dk for those nodes with observations, where k is the observation index in the data vector d. Reconstruction of the three dimensional surfaces from the images is an ill-posed problem because some information is lost during the imaging process (projecting 3-D onto 2-D) (Poggio et al., 1985). Other reasons are the noise and erroneous, inconsistent, and sparse measurements (Terzopoulos, 1983). Regularization is the frame within which an ill-posed problem is changed into a well-posed one. A problem is well-posed if a solution exists, is unique, and depends continuously on the initial data. It must also be well conditioned to ensure numerical stability (i.e., robust against noise) (Poggio et al., 1985). By means of regularization, and having suitable a priori knowledge, a valid solution is determined by reformulating the ill-posed problem as minimizing an energy function E that comprises two functionals;
E(s) = D(s) + λ2 S(s).
(2)
The first functional, D, provides a measure of the closeness of the solution (s) to the observations. It is usually taken to be the sum of the square of the differences between interpolated values and the original data. The second functional, S, measures the smoothness of the solution as compared to a specific surface spline. The variable λ2 is the regularization parameter that controls the influence of the two functionals. If λ2 is very large, the second functional dominates and turns the solution into an interpolation (close to data). On the other hand, if λ2 is small, the solution stresses the smoothness of the surface.
3 SURFACE DISCONTINUITY DETECTION Almost all methods for surface interpolation assume a smooth surface. Thus, when they encounter a discontinuity in the surface, the interpolated surface will exhibit an unrealistic behavior, manifested as oscillation and overshooting. Hence, discontinuities must be detected at a certain phase of the interpolation process. Additional interest arises from the fact that a discontinuity in the surface signifies a feature that may be interesting for mapping applications. A crease (i.e., change of surface orientation) may imply a roof of a building, or a valley. Similarly, a step (i.e., depth discontinuity) may represent a defining wall of a building, or occlusion. Only a limited number of methods for detecting discontinuities in the surface have been proposed and implemented. Some of these methods identify discontinuities prior to the interpolation process (e.g., Grimson and Pavlidis, 1985), while others detect discontinuities subsequently (e.g., Terzopoulos, 1983). Nevertheless, methods based on the “line process” concept allow simultaneous execution of interpolation and detection. Geman and Geman (1984), first to introduce this concept, models the image (or surface) as a sample function of a stochastic process over a set of neighboring pixels. The concept of “line process” is then added to the estimation problem. The line process is a set of unobserved attributes located between the pixels and depicts the geometry of the boundaries between them. In the context of the energy model, the purpose of the line process is to disjoin bordering pixels and reduce the total energy if the values of these pixels are different. Available knowledge about the existence of the boundaries at specific locations can be accommodated by pre-setting the variable of the line process associated with these pixels to one, otherwise it is set to zero. Eventually, the lowest energy state will result by breaking the surface into small pieces around each data point. To avoid this, a penalty α should be paid (in terms of energy) when a break line is introduced. Thus, a break line will only be introduced when paying the penalty is less expensive than not having the break line at all. The penalty function takes the form P = α li, where α is a scalar, li is the line process at point i. When the penalty is added to the original energy function (2), the problem is changed into minimization of E = D + λ2 S (1-l) + P.
(3)
The result is a combination of a continuous function for the surface and a discrete one for the lines. This combination
allows surface reconstruction and discontinuity detection at the same time. However, E is a nonconvex function that has many local minima, only one of which is the global one, which denotes the sought solution. Straightforward application of the minimization techniques cannot be applied; instead, special techniques must be followed. One approach will be discussed in the next section.
That would reduce the problem into minimization of E over {ui,j}, and simplifies the computations. Elimination of the line variable can be accomplished by two steps, the first one combines the two terms S and P and replaces them by hα,λ as in the following hα,λ(t, l) = λ2 t2 (1-l) + α l.
4 SOLUTION OF NONCONVEX FUNCTIONS The “Graduated Non-Convexity” (GNC) algorithm, proposed in (Blake and Zisserman, 1987), is a deterministic approach for the solution of the nonconvex function (equation 3). According to this approach, the line process P is merged with the interpolation functional S. The result is a function that controls the interaction between neighboring grid cells. Such a function prefers continuity in the surface, but allows occasional discontinuities if that makes for a simpler overall description; a theme called “weak continuity constraints.” This section provides some details on the method as presented in (Blake and Zisserman, 1987).
This function describes the local interactions between neighboring sites. More specifically, when there is a discontinuity, then l=1 and hα,λ= α, otherwise hα,λ= λ2 t2. The two parts of this function can be seen in Figure 1. The energy E for the weak membrane then becomes E = D + ∑ hα ,λ ( ui , j − ui − 1 , j , ni , j ) + ∑ hα ,λ ( ui , j − ui , j + 1 , mi , j ) (8) i. j
i,j
hαλ
The behavior of the membrane would be defined by the same three-component energy function of equation (3). The functional D measures the faithfulness of the solution u to the data d. Using the physical analogy, it represents the energy of the springs that tie the membrane to the data. The second component, S, is basically the energy of the membrane augmented with the line process, which would break the membrane if the tension is too high. The component P is the sum of penalties α paid for each break in the membrane. This arrangement facilitates the principle of the weak continuity constraints. In other words, the membrane is constrained to be continuous, however, it is allowed to be broken at a point in the data when this situation reduces the total energy. At such a point, the associated line process is set to 1. To extend the line process concept to the case of the membrane, Blake and Zisserman adopt a special arrangement for the line variables that facilitates the construction of a convex approximation of E. The line process consists of two sets of variables; ni,j represents a discontinuity between two nodes northerly connected, and mi,j represents a discontinuity between two nodes easterly connected. The discrete representations for all three components are
∑(u
− d i , j )2
∑(u
− ui − 1 , j )2 ( 1 − ni , j ) + ∑ ( ui , j − ui , j + 1 )2 ( 1 − mi , j ) (5)
i, j
l=0
α
4.1 The weak membrane
D=
(7)
−
l=1
t
α λ
α λ
Figure 1: The energy function for the local interaction The second step in the minimization of E is based on the fact that D does not involve the line process. Hence, the minimization operation can be redistributed, and hα,λ alone would be minimized over li,j. Thereafter, the line process would have been eliminated from the energy function, and a new and simpler cost function F is generated. The problem becomes one of minimizing F with respect to ui,j, where F = D + ∑ gα ,λ ( ui , j − ui − 1, j ) + ∑ gα ,λ ( ui , j − ui , j + 1 ) . i,j
(9)
i,j
The function gα,λ(t) is the neighbor interaction function. Figure 2 shows clearly that gα,λ(t) is the minimum of the two components of hα,λ. Mathematically, it is expressed as λ2 t 2 gαλ = α
if t < α / λ otherwise
(10)
gαλ
(4)
i, j
S=
i, j
i. j
P = α ∑ li , j
i, j
α
(6)
i, j
The factor λ2 has now the physical connotation of elasticity or stretchability of the membrane. The total energy function now consists of a set of the real-valued variables {ui,j} and of a set of Boolean line process {li,j}. The problem now becomes of minimizing the energy function over these two sets of variables. However, minimization of E over the line process can be done in advance (i.e., eliminating the line variables).
−
α λ
t α λ
Figure 2: The neighbor interaction function
If discontinuities are known in advance, the line variables are turned “on” (i.e., nij = 1, or mij = 1, whatever the case may be) at all points along the contour. On the other end, after achieving the solution, the line process li,j can be explicitly recovered by using the function g(t). The line variable would be set to 1 (i.e., a breakline) if the absolute difference between two neighboring nodes is larger than ( α λ ), otherwise, it is 0 (no break). 4.2 Graduated solution of nonconvex energy Convexity is a sufficient condition for all local minima to be a global minimum. However, the function F is nonconvex since it may have many local minima, a situation created by the binary nature of the line process. This means a naive gradient descent will stick at a local minimum that is of higher cost than the global minimum. Besides, which specific local minima would be reached depends mainly on the starting point of the descent. Blake and Zisserman (1987) developed a technique for solving the nonconvex function, named Graduated Non-Convexity algorithm (GNC). It is based on the gradual solution of a sequence of nonconvex functions, each of which is approximated by a convex function. The final solution represents a convex approximation that encompasses the original function and has only one global minimum. The nonconvex function F is gradually approximated by a convex one through a family of p intermediate functions; F(p). The nonconvexity parameter p represents a sequence of real numbers ranging from one to zero. The function F(p) to be minimized has the general form of F ( p ) = D + ∑ g ( p ) ( u ).
definite, hence, F(1)(u) is a convex function with a unique minimum. Then from that minimum, the local minimum of each F(p) is tracked continuously as p varies from 1 to 0. As p goes to zero, F(p) becomes closer to the original nonconvex one. In practice, a sequence of discrete values of p is used. Each F(i) is minimized, using the minimum of the previous F(i-1) as a starting point. The claim is that following such a progression is more effective, especially when λ is large, than optimizing one function F. Numerical solution of the function sequence F(p) is achieved by using gradient descent (or relaxation) algorithm. The authors in (Blake and Zisserman, 1987) propose the method of successive over-relaxation (SOR) as the iterative minimization algorithm. These methods would be further discussed in a later section. The principal equations of the algorithm, (11), (12) and (13) are functions of the parameters λ and α. Although these parameters are crucial for the success of surface interpolation and discontinuity detection, there is no clear or exact principle for defining them. However, the principle in using the weak membrane approach is to adopt the lowest energy configuration. Thus, comparing the energy of the continuous membrane with that of a surface with different types of discontinuities would determine the effect of λ and α on the performance of the algorithm. The authors set five different aspects of performance in discontinuity detection that depend on these parameters (Blake and Zisserman, 1987), namely • Contrast Sensitivity Threshold, ho. It determines the minimum contrast for detecting an isolated step edge. Accordingly, a step will be detected if its height h is larger than ho = 2α / λ . •
(11)
The neighbor interaction function is also modified accordingly into a function of λ, α, and p. The function g is approximated be a set of splines (as shown in Figure 3), denoted as g(p), and expressed as
•
λ2 t 2 if | t| < q ( p) 2 gαλ ( t ) = α − c( t − r ) / 2 if q ≤ t < r α if | t| ≥ r where c* 2 α 2 1 c= , r = α + 2 , and q = 2 . c λ λr p
(12) •
(13)
g αλ
g ( p) αλ
•
Characteristic Length, or Scale ( λ). It is also a distance that defines the interaction between discontinuities; if two discontinuities are separated by distances larger than λ, then they do not interfere with each other and are considered independently by the algorithm. Gradient Limit, gl. A ramp of gradient g will be detected as a step when g > gl = ho /2λ. A steep ramp with g much larger than gl will be detected as multiple steps since this solution has less energy. Immunity to Noise (α). No spurious discontinuities are generated provided that α > 2σ2, where σ2 is the variance of the mean noise. When the difference in the values of adjacent pixels exceeds ho = 2 λ , then double edges occur.
α
t q
r
Figure 3: Spline approximation of g The factor c* is equal to 0.25 for the membrane. At the start of the algorithm, p = 1 and c = c*. The function F(1) is a crude approximation to the nonconvex function. However, the authors prove that the Hessian matrix of F(1) is positive
These parameters provide a guideline for defining the values of λ and α. Based on the broad expectations of the surface, one may choose the values for λ and α with the guidance, and the confinement, of mainly the desired height of a step to be detected ho, and the gradient limit gl. Selecting a larger λ reduces the sensitivity ho, hence, more discontinuities will be detected. On the other hand, it will also mark false steps due to noise. A larger λ also reduces the gradient limit resulting in spurious multiple steps. 4.3 Weak membrane on sparse data By its nature, when there is a need for breaking a surface between two nodes to reduce the energy, the algorithm places a break line between these nodes. However, when dealing with sparse data, two data points may be several grid nodes apart. In
this case, disjoining any two grid nodes amid these two points will yield the same low energy. Thus, there might be an ambiguity in the exact location of a discontinuity. Because of that, direct reconstruction from sparse data is not recommended. Instead, the sparse data is first converted to a dense one using a continuous membrane (Blake and Zisserman, 1987). 5 MULTIGRID IMPLEMENTATION The method of Graduated Non-Convexity solution of the weak membrane has the virtue of the simultaneous execution of surface interpolation and discontinuity detection. However, it would perform better if a dense representation is first produced from the sparse data. Accordingly, the development of a method for surface interpolation and analysis procedures involves two main sub-tasks. The first one is surface densification that produces a preliminary dense representation of the data. The second task is the actual combined process of interpolation and discontinuity detection by the weak membrane. Both tasks involve the application of a relaxation method. 5.1 Relaxation methods Numerical solution methods for elliptic partial differential equations are classified into two classes: direct and iterative. Direct methods, such as Gauss elimination, LU-decomposition, and Fast Fourier Transform are faster, non-iterative, and more robust than the iterative ones. They are, however, feasible only when the number of grid points is very small, the coefficients of the difference equations have a special and simple form, and a computer with a huge memory space is available (Nakamura, 1991). Iterative methods, on the other hand, can be used for any form and size of a problem. These methods, also called relaxation methods, basically compute the value at a point based on the values of its four neighbors (Jacobi method). The procedure is repeated until the computed values show little further changes. Acceleration of the rate of convergence can be obtained by using the updated values in the computing of the next point immediately as they become available (Gauss-Sidel method). Further acceleration of convergence can be obtained by over-correcting the solution at any iteration (Successive Over-relaxation method). Standard iterative methods are very effective at eliminating the high-frequency components of the solution, while leaving the low-frequency components relatively unchanged. As a result, relaxation methods are inherently inefficient in propagating information over large areas of no data. 5.2 Multigrid methods Special techniques must be developed in order to treat the deficiency of relaxation methods in dealing with low-frequency components, and speed up the convergence. The method of multigrid provides the remedy for this problem (Briggs, 1987). It provides two strategies; the coarse grid correction and the nested iteration, both of which are to be realized within a multigrid representation of the problem. The principle of the coarse grid correction scheme is to transfer (restrict) the solution to a coarser grid when relaxation begins to stall (due to reaching smooth error modes). The low frequencies on a fine grid look less smooth (i.e., become
higher) ones on a coarser grid. The nested iteration scheme prescribes that the initial approximation for a fine grid is obtained by interpolating (prolonging) the solution of the coarser grid, instead of starting with any arbitrary approximation. The solution of the coarser grid, in turn, is also found by the same process from even coarser grids. The solution of the coarsest level is obtained by exact, or more rigors, solution of the problem. With such an approach, there is a smaller number of unknowns to update, smaller number of relaxation sweeps, and, thus, faster rate of convergence. Nested iteration is then a recursive application of the correction scheme. Implementing the tasks of surface densification and the weak membrane within a multigrid scheme will, therefore, improve their performance. Additionally, such a scheme provides the means for communication between the different levels of the image pyramid, and between the different phases of the hierarchical surface reconstruction. Nevertheless, hierarchy is an intrinsic concept of surface reconstruction. 5.3 Multigrid surface densification by a membrane According to the multigrid approach, 1) The original data must first be preprocessed and resampled according to the definitions of the grid at each level. That would increase the percentage of nodes with data versus nodes without data at the coarser grids. 2) At the grid of the lowest resolution, the solution is obtained by the repeated passage of the membrane until there is no change in grid values. 3) The resulted surface will then be prolonged into higher resolution to serve as the initial surface. The corresponding data set will be introduced into the surface as boundary conditions. 4) On the current level, one relaxation sweep is performed, at the end of which, convergence is tested. This step would be repeated as long convergence is satisfactory. When the solution is reached, then the process proceeds to the next higher level (back to step 3). Otherwise, when convergence is slow, residuals would be transferred to coarse grid and step 4 is started again. As the result of following this approach, gaps representing the low frequencies in the surface are filled, and a better approximation is produced for the processes at the next level of image resolution. One must differentiate between image resolution and surface grid resolution. For every image resolution there is a surface representation of the same resolution in the object space. However, the surface is further sampled into grids of coarser resolution to satisfy computational requirements. This is more the case for the lowest resolution of the image pyramid, since there is no good initial approximation for the surface to start with. 5.4 Multigrid GNC on weak membrane The GNC algorithm is incorporated in the multigrid scheme. At any specific level of the multigrid representation, the processes starts with a prolonged solution of the previous grid as well as the corresponding data. A complete cycle of the multigrid membrane would be performed. At the end of such cycle, the GNC routine would be activated. The results of performing the GNC algorithm are a smoothed surface and a
set of detected discontinuities. These results would be used to provide an approximation for the new cycle at the next finer surface grid. With regard to the GNC parameters, adopting the multigrid approach has an additional benefit of assisting their estimation. As was mentioned earlier, the performance of the GNC algorithm is controlled by five aspects, only two of which are independent and required to be specified. This study chooses λ (the scale) as one of these parameters. Its value is determined in conjunction with the level of surface grid. At the lowest resolution of the grid, it is chosen somewhat larger than one grid cell (e.g., 1.1) to keep the smoothing and breaking of the surface at the minimum. As the algorithm progress along the hierarchy, λ is increased by one. There is still one more parameter to be defined. The contrast threshold ho has an understandable physical connotation as it determines the minimum contrast for the detection of an isolated step edge. However, ho can not be related directly to the height of the smallest step desired to be detected. After several experiments, this study found that GNC performs better when ho is close to twice the mean deviation from the mean of the surface. This way, ho is related to the surface data and its variations.
The three-dimensional data points are relayed to the surface interpolation phase. They represent mainly points on successfully matched edge segments in the corresponding image resolution. Figure 5a represents the first set of data points at the 1k×1k image resolution, while the second set of data is depicted by Figure 5b. The first set of points will be used mainly for investigating the two tasks of surface interpolation, namely surface densification and GNC on the weak membrane. The second set will be used to extend the findings to the multiresolution multigrid implementation.
5.5 Transferring discontinuities One goal of this study is to incorporate breaklines into the interpolation process whenever they become available. To realize this goal, the structure of the array that represents the surface must be modified to accommodate the line process variables beside the surface value at each grid node. Additionally, the prolongation and the restriction operators of the multigrid procedure must also be modified in order not to include data across break lines.
Figure 4: The OSU Library section of the digital image
One conceptual issue arises at this junction. The detected breaklines are located between grid cells, thus their definitions are related to the resolution of the grid. In a multigrid structure, each grid cell is represented by four cells at the next higher level. The question is then how strong must the algorithm enforce detected discontinuities in the finer grid? This study takes a conservative action in this matter. The detected discontinuities will affect only the prolongation of the coarse grid. They will not be enforced during the interpolation process. This way, the algorithm allows refining and redefining of the discontinuities with better accuracy. Thus, in the current implementation, discontinuities detected in one level would influence surface approximation for the next higher level, but will not be retained explicitly.
6 TEST DATA AND EXPERIMENTS The experiments are conducted using a real data set. The set represents a portion of the campus of The Ohio State University, specifically the Main Library (shown in Figure 4). An image pyramid is constructed according to the hierarchical surface reconstruction approach. The data used for the experiments is related to two image resolutions: a coarse one of a 1k×1k image size, and a finer resolution corresponding to 2k×2k image size. These resolutions correspond, respectively, to a 2-meter and 1-meter grid resolution in the object space.
(a) Matched edges at the 2-meter resolution of the image
membrane. Figure 8 shows the detected breaklines at that level.
(b) Matched edges at the 1-meter resolution of the image Figure 5: The OSU Library test data The first set of experiments is to compare direct application of a membrane (single grid) with the multigrid implementation. The data that correspond to the 1k×1k image resolution is first resampled into three levels of a cell size equals 2, 4, and 8 meters respectively. The criterion used to determine the number of grids is the percentage of the nodes with data versus nodes without data (50% or more). Each sampled set is read into the initial approximation of the solution in the beginning of the process at the corresponding level. At the end of the multigrid interpolation, the surface would be represented by three grids of different sizes with the accuracy of the finest one. Figure 6 shows the surface representation at the end of the interpolation by multigrid membrane.
Figure 7: The surface for the 2-meter image resolution The last set of experiment is to apply the algorithm on multiresolution data. The data used are those obtained by the matching procedure on the 2k×2k image resolution. The surface and discontinuities that became available on the previous (coarser) level are used to aid the interpolation on level. The top of Figure 9 shows the surface as interpolated by continuous membrane. The bottom surface of Figure 9 is the surface as interpolated by the multigrid implementation of the weak membrane.
Figure 6: Surface by multigrid membrane at 2-meter resolution The direct implementation of the membrane (i.e., single-grid relaxation) is applied to the same data of the finest grid. The final surface produced by the multigrid approach is slightly better (visually) than the one produced by this direct application of the membrane. The difference is mainly in the areas of a large span between data, and it would be even greater if the data points were more sparse and less in number. The second set of experiments is concerned with the intermixed implementation of the membrane and the weak membrane. The same previous set of data (at three levels) is used for these experiments. Figure 7 represents the resulting surface in the final level of the OSU Library data after of a complete multigrid cycle of the continuous and the weak
Figure 8: The detected discontinuities for the 2-meter resolution Two remarks worth mentioning here. The first remark is that for clarity and comparison sake, the surface depicted by Figure 8 is sampled at 2-meter grid cell although the data is of 1meter resolution. The other remark is that, if the proposed method is fully implemented, then the collection data on the fine resolution must be guided by the findings on the previous one. This, however, is not the case for this set of data.
the input for this subsystem is a set of sparse points, the surface must first be densified using a multigrid implementation of the membrane. The multigrid implementation is superior to the single grid GNC because it provides a better densification for the surface. Moreover, it lends itself to a more objective definition of the required control parameters (namely, the scale, λ, and the contrast sensitivity threshold, ho). In addition, the multigrid representation of the line process provides the mechanism to communicate information on detected breaklines with other modules in the surface reconstruction system. The conducted experiments proved that this algorithm is suitable and applicable for dealing with large-scale images of urban areas. Furthermore, it fits perfectly in the hierarchical multiresolution surface reconstruction paradigm.
Figure 9: The surface and the detected discontinuities for the data of the 1-meter image resolution
7 CONCLUDING REMARKS •
The purpose of this study is to develop a method for interpolating the surface from a set of sparse points obtained from matching aerial images. The solution to this problem is of great interest in stereo matching and surface reconstruction. This study elected to work within the scheme of hierarchical multiresolution surface reconstruction in the object space, developed by Schenk et al. (1990). However, the method is general and can be employed in any other reconstruction system. Furthermore, the method would also be particularly useful for interpolating topographic or non-topographic surfaces from any type of surface observations.
Another concern for this study is defining the ways for detecting and treating surface discontinuities. This interest is induced by the fact that the performance of any interpolation method will suffer greatly without incorporating such discontinuities. This is particularly true for large-scale urban imagery. Additionally, surface discontinuities are definite indications of boundaries of objects on the topographic surface (e.g., buildings), specially in urban scenes. These boundaries are crucial for the higher level vision tasks such as object recognition. The main contribution of this study is the development of an interpolation and breaklines detection subsystem within the framework of a surface reconstruction system. This study proposes a multigrid implementation of the Graduated NonConvexity (GNC) algorithm for the weak membrane. The algorithm presented by this study offers the solution for the problems associated with the original GNC algorithm. Because
The main limitation of the method is that locating the discontinuities is not directly related to the data when there is a wide span between two data sets. In such a situation, breaking the surface (if required) will produce the same low energy anywhere between them. It is logical that no algorithm would provide information when there is not enough data. However, detected discontinuities by this algorithm can be considered as hypotheses on breaklines. They may be compared or incorporated with other hypotheses, one of which is the set of edges in the image. There are several ways to benefit from the detected discontinuities; some of which may be based on • Using these potential discontinuities to guide the matching process at the next level of image resolution. The matching operator would be required to confirm or reject these hypotheses by examining the image at the locations defined by the detected discontinuities.
•
•
Changing the Boolean line process into a real-valued variable. The value of a line process associated with a discontinuity in the image or in the surface may set close to zero in the beginning, and then increased each time a discontinuity appears at that place. Accordingly, the higher the value, the more certain the discontinuity. Studying, at the end of each cycle, the scale-space behavior (finger print) of the discontinuities in the surface and in the image. Such a study may adopt either a deterministic, probabilistic, or knowledge-based approach.
REFERENCES Al-Tahir, R., 1996. Interpolation and Analysis in Hierarchical Surface Reconstruction. Report 435, Department of Geodetic Science and Surveying, The Ohio State University, OH, 88p. Al-Tahir, R. and Schenk, A., 1992. On the interpolation problem of automated surface reconstruction. Proceedings of the 17th Congress of the International Society for Photogrammetry and Remote Sensing, 29(3):227-232 Blake, A. and Zisserman, A., 1987. Visual Reconstruction, MIT Press, MA, 225p.
Briggs, W., 1987. A Multigrid Tutorial. SIAM, Lancaster, Pennsylvania, 90p. Geman, S. and Geman, D., 1984. Stochastic relaxation, Gibbs distributions, and the Bayesian restoration of images. IEEE Transactions on Pattern Analysis and Machine Intelligence, PAMI-6(6):721-741. Grimson, W. and Pavlidis, T., 1985. Discontinuity detection for visual surface reconstruction. Computer Vision, Graphics, and Image Processing, 30. Nakamura, S., 1991. Applied Numerical Methods with Software, Printice Hall, NJ. Poggio, T., Torre, V. and Koch, C., 1985. Computational vision and regularization theory. Nature, pp. 317:314-319. Schenk, A. and Toth, C., 1991. Reconstructing visible surfaces. Industrial Vision Metrology, SPIE, pp. 1526:78-89 Schenk, A., Li, J-C. and Toth, C., 1990. Hierarchical approach to reconstruct surfaces by using iteratively rectified imagery. Proceedings of the International Society for Photogrammetry and Remote Sensing Symposium, 28(5/1):464-470. Terzopoulos, D., 1983. Multilevel computational processes for visual surface reconstruction. Computer Vision, Graphics, and Image Processing, 24(1):52-96.
