MULTI-RESOLUTION QUALITY IMPROVEMENT FOR PATCH-BASED TEXTURE SYNTHESIS IN WAVELET DOMAIN Fan Gu , Yuri Rzhanov Center for Coastal and Ocean Mapping Alejo Hausner Department of Computer Science University of New Hampshire, NH, USA {fangu,yuri.rzhanov}@ccom.unh.edu,
[email protected]
ABSTRACT Patch-based texture synthesis has been widely used in computer graphics and computer vision due to simplicity and efficiency. The basic idea is to add one patch of the sample texture at a time to the target texture. Much work has been done to minimize the appearance of seams in the overlap region of two adjacent patches. Among various approaches to blending, stitching two patches along the minimum error boundary cut is considered to be one of the most efficient. However, in real textures, especially the semi -stochastic ones, perfect cuts do not exist; therefore, the defects due to the inconsistency between the patches become an important problem for the synthesized texture quality. In our paper, two novel models that have not been considered in the previous approaches are proposed to overcome these difficulties. We show that the texture quality can be improved by performing multi-resolution operations on the overlap region of adjacent patches in the wavelet domain.
relatively small ones. For these scenarios, patch-based texture synthesis was proposed [4] and further explored in terms of speed acceleration and accuracy. However, quality is still a problem in these algorithms due to the complexity of texture: visible seam between two adjacent patches is negatively affecting the texture quality. In our paper, by taking advantages of the multi-resolution and space-frequency locality properties of the wavelet transformation, two novel models are proposed for the purpose of eliminating defects and enhancing texture quality. The paper is organized as follows: second section reviews the previous work on texture synthesis, blending techniques, and wavelet application in texture related areas; in the third section, the processes of patch-based texture synthesis are reviewed and our two models are detailed; results and conclusions are stated in the fourth and fifth sections, respectively.
KEY WORDS Texture mapping, texture synthesis, wavelet domain, multi-resolution, blending
2. Previous Work 1. Introduction
This section gives an overview of the previous work and is divided into three parts: the category of texture synthesis techniques and their improvement algorithms, blending techniques used in texture synthesis and other potential methods that were implemented in image processing, and applications of wavelet transform algorithms in texture related research.
Technology and techniques for producing high-quality synthesized images have attracted much attention in recent years. From special effects in films on the big screen to realistic virtual worlds in video games on the small screen, there is a continuous demand for synthesized images of increasingly higher quality. One of the most popular techniques that have been adopted to meet this demand is texture mapping [1]. In contrast to a three-dimensional reconstruction approach, the texture mapping approach generates a novel view from a smaller sample of the “real-world” and makes this view more realistic by mapping the appropriate visual texture onto the scene objects. To be successful, however, this technique requires arbitrarily large patches of texture. This, in turn, drives the need for better and faster techniques for synthesizing large texture patches from
534-248
2.1 Texture Synthesis In the recent years, several approaches have been proposed for texture synthesis. They can be divided into two categories: pixel-based and patch-based algorithms. Usually, pixel-based approaches take up to several hours of computations to produce impressive results. Some accelerated methods were proposed by Hertzmann et al. [2] and Wei and Levoy [3] that use multi-resolution and
266
Tree Vector Quantization (TVQ) techniques, however, these methods are complex to implement.
transform. Bar-Joseph and colleagues [14] also worked with texture synthesis in the wavelet domain. In their research, the input textures are treated as sample signals generated by a stochastic process. In the application of texture classification, Arivazhagan and Ganesan [15] described metrics to compare an input texture with several samples stored in a database, using wavelet statistical features and wavelet co-occurrence feature.
Patch-based texture synthesis algorithms are working by joining the patches together, “quilting” them, and making sure that they fit well. Implementations include the simple and generic image quilting algorithm proposed by Efros and Freeman [4], the efficient graph-cut algorithm [5] and the smart-copy algorithm [6]. These techniques are much more efficient than pixel-based approach since the texture is built at a much coarser scale, while being able to keep high frequencies of the sample. In [7], real-time generation of large textures using special data structures and optimization techniques is reported. Our work is an extension of patch-based texture synthesis.
The blending technique realized in wavelet domain has also shown its effectiveness. Iddo and Dani [11] gave a demonstration of this application. However, their algorithm of combining coefficients of sub-band to compensate for illumination difference is only applicable when multiple images are highly matched. Blurring effects could occur if images do not match. In the situation of texture synthesis, even the most effective sample selection algorithm cannot guarantee that exact matching could be achieved.
2.2 Blending In order to achieve high-quality images from two or more image sources, the technique of blending plays an important part. The main blending approaches reported in the literature include: feathering, multi-resolution splining [8], image fusion [9], gradient domain stitching [10], and wavelet blending [11].
3. Patch-based Texture Synthesis In patch-based texture synthesis, small square patches from the original sample image are combined to form the final texture. The approach starts by randomly picking a patch B0 and positions it at the top-left corner of the output texture. The dimension of patches DB is user-defined and intuitively it should be the size of the main texture elements or texels present in the sample image. In the experiments, using a patch of dimension between half and a quarter of the size of the original sample works well. The synthesis process follows then by adding patches side by side and once a full row is completed, the process continues for the row below, and so on. Three possible configurations for overlap zone matching are illustrated in Figure 1. For each patch, there is an overlap zone also with a user-defined width DE . The optimal dimension of DE depends on the texture being generated. If it is too small, details of the texture content will not be captured; on the other hand, if chosen too large, it will negatively impact the algorithm’s performance; for example, the appearance of blocks and the synthesizing speed. In our experiments, we chose a quarter or half of the patch size.
The minimum-error boundary cut algorithm has been used to achieve a better transition between the adjacent patches. In this approach, a minimum-error boundary cut across the overlap region of two adjacent patches is found and used to stitch patches [4]. An improvement to the algorithm is the graph-cut technique [5] which deals with more complex shapes of overlap region. This method produces near-perfect results when two adjacent images match well enough and we can find a seam that is invisible after the stitching. However, in the real world textures, most of the time, this ideal situation rarely occurs and defects due to the inconsistency of adjacent patches damage the quality of synthesized images. Thus, further explorations are necessary for this situation.
2.3 Wavelets Since wavelets were first introduced in the graphics community [12], they have come a long way and are now an important tool in many graphics and image processing applications. Performing operations in the wavelet domain has the following advantages [11]: • Sparse representation • Multi-resolution • Progressive computation • Space-frequency locality • Compatibility with emerging standards
B0
Bk
DB Bk
DE Overlap Zones
Bk Boundary Zones
Boundary Zones
Fi gure 1. Illustration of patch -based texture synthesis adapted from [4]. First configuration or starting point (left), second configuration (middle) and third configuration or L-shaped matching (right). Darker grey area represents texture already synthesized.
Methods based on wavelets for texture synthesis, analysis and classifications have also appeared. Portilla and Simoncelli [13] proposed a statistical model for texture representation using an over-complete complex wavelet
267
3.1 Patch Selection An illustration of the stitching process is shown in Figure 2. The operation on the boundary zones and mask matrices is defined as multiplying corresponding pixels and mask values.
The critical part of the algorithm is the selection of the next patch Bk to be pasted on the texture being constructed. The RGB metric is used to compare patches and build a list of candidate patches that satisfies an error criterion at the border area. Then, a random patch is selected from this list. To build this list, the input sample is searched for all possible patches. If there is no patch satisfying the condition, the algorithm picks the patch with the smallest difference. Two texture patches of the same size and shape are considered to be matching if the following is satisfied:
D ( Z i , Z i +1 ) < δ
I1
M1 Output
(1)
I2
M2
Figure 2. Illustration of the stitching process. Input I1 and I2 represent the boundary zones of two adjacent patches (here is an example of vertical overlap path). M1 and M2 are two stitching mask matrices in which black stands for “0” and white stands for “1”.
Here, Zi , Zi+1 denote the boundary zones of two overlap patches, while D(*) is the mean square root of pixel values difference in three color channels of two adjacent patches and δ is a pre-defined constant.
3.2 Calculate the error surface
4. Wavelet -based multi-resolution operations
Before manipulating with the overlap region of the new patch and the constructed patch, an error surface is calculated. Different cost functions will result in different error surfaces; typically used metrics are L2 -norm and squared root difference. In our experiment, the squared root difference metric is used on three color channels, that is:
The wavelet transform (WT) is a mathematical tool that can be used to describe 1D or 2D signals in multiple resolutions. A wavelet transform is obtained through a sequence of low-pass and high-pass filters, alternated with down-samplings. The result of the wavelet transform is a down-sampled smoothed signal and several detail coefficients obtained at each down-samp ling. In other words, the wavelet transform produces a signal that encodes both information on the original signal values and its multi-scale edges. Our algorithms are based on the space-frequency locality and multi-resolution properties of the wavelet.
eij = (1R ij −2R ij )2 +(1G ij −2G ij )2 +(1B ij −2B ij )2
(3)
Here, R, G, B represent values of three color channels, i and j are two dimensional indices for two boundary zones and the super-script indicates the patch number.
The ideas of our algorithms are coming from the basic principle of multi-resolution spline blending [8]; that is, performing multiple operations on different frequency sub-bands to smooth the transition zones. It is consistent to the rules of Human Visual System that human eyes are more sensitive to the sudden changes of the image. The difference here is that our models are wavelet based while [8] is using the Laplacian pyramid. When the process comes to the transition zone, the technique described in [8] used the method of feathering on the overlap region in different sub-band, which is not practical in our case because of the occurrence of blurring. We have improved it by taking the advantages of minimum-cost boundary cut in sub-bands. Results show that our methods improve the overall texture quality effectively.
3.3 Determine minimum-cost boundary cut Once the error surface is constructed, the minimum-cost boundary cut algorithm is applied to it. Define a cumulative path cost E based on the pixel error values eij on the error surface. Specifically, for a vertical portion of the overlap patch, E is calculated from a pixel in the first row to the current pixel recursively in terms of row index i and column index j as follows:
eij Eij = eij + min( Ei −1, j −1 , Ei−1, j , Ei −1, j +1 )
i =1 (4) i >1
As stated in the beginning, our method is an expansion of patch-based texture synthesis. The basic procedures are the same as [4] which have been detailed in section 3. In this section, we will only focus on the operation of boundary zones. It should be noted that, in calculating cost functions and error surfaces, the values are from pixel values in the first model and wavelet coefficients in
When the appropriate arguments to the min imum function are obtained at the edges of the overlap patch, the minimum error boundary cut can then be determined by searching for the last row of the overlap patch, identifying the minimum value and tracing backwards. The similar formula can be applied to the horizontal overlap path.
268
the second model. In addition, due to the dimension sizes that are relatively small, our models are tested on one level of wavelet transformation.
3.
4. 4.1 Model I
5.
When applying one level wavelet transformation to 2D image signal, four sub-bands containing different frequency information are generated: low-low (LL), high-low (HL), low-high (LH), and high-high (HH). The sub-band of LL has the least detailed information; however, changes to this frequency domain will affect the overall image resolution. So it is necessary to use a wide blend mask to decrease the transition artefacts. For the mid-frequency sub-bands (HL and LH), the same blend mask is applied for consistency. In order to further smooth the transition zone, we set the mask values for HH sub-band to zero.
6.
The new mask is obtained by setting the values in position of LL, LH, HL as the result of Gaussian pyramid and that of HH to zero. Obtain the wavelet coefficients of two input boundary zones. Apply masks to the input boundary zones to obtain the output matrix. Inversely transform the output matrix to obtain the final synthesized zone.
4.2 Model II A simpler but efficient enough model can be achieved by directly calculating the mask fro m the LL sub-band. In other words, obtain the error surface from the LL sub-band and operate the minimum-cut boundary cut to the three lower frequency domains. As in the Model I, the HH sub-band mask is set to zero for the purpose of smoothness. The scheme of the algorithm is shown in Figure 4:
The blending mask is derived from the Gaussian pyramids of the special minimum-cost boundary cut. In this model, we used the Haar wavelet because of its simplicity and size consistency with the pyramids. The scheme of the algorithm is illustrated in Figure 3.
I1 DWT
I1
M1
M1
DWT
Output
IDWT
IDWT
DWT I2
DWT M2
M2
I2
Figure 3. Illustration of stitching process in Model I. Input I1 and I2 represent the boundary zones of two adjacent patches (here is an example of vertical overlap path). M1 and M2 are two stitching mask matrices in which black stands for “0” and white stands for “1”. In this model, the Haar wavelet is used for the wavelet transform. DWT stands for the discrete wavelet transform and IDWT stands for the inverse discrete wavelet transform.
Figure 4. Illustration of stitching process in Model II. Input I1 and I2 represent the boundary zones of two adjacent patches (here is an example of vertical overlap path). M1 and M2 are two stitching mask matrices in which black stands for “0” and white stands for “1”. In this model, the forth order Coiflet wavelet is used for the wavelet transform .
The process steps are detailed as follows:
The process steps are detailed as follows:
1.
2.
1.
Find the minimum-cost boundary cut for the input boundary zones I1 and I2 and calculate mask matrices M1 and M2 . Calculate the Gaussian pyramid for these two mask matrices.
2.
269
Obtain the wavelet coefficients of two input boundary zones. In the sub-band LL, find the minimum-cost boundary cut for the wavelet coefficients of boundary zones and calculate mask matrices M1 and M2, respectively.
3. 4.
The new mask is obtained by copying values in the position of LL to that of LH, HL. Inversely transform the output matrix to obtain the final synthesized zone.
In this model, however, the choice of wavelet transform has stronger influence on the texture quality. In evaluating the wavelet functions in image processing, the following properties are considered [16]: • Compact support: qualifying both time and frequency localizations; • Symmetry: avoiding dephasing; • Orthogonality: allowing fast algorithms; • Regularity: useful for getting nice features, like smoothness of the reconstructed signal or image; • The number of vanishing moments: relating to filter order or filter length.
(a)
(c)
Typically used wavelets are the Haar Wavelet, the Daubechies Wavelet, and the Coiflet Wavelet. In our experiments, the forth order Coiflet Wavelet which bears the characteristics of arbitrary regularity, compactly supported orthogonal, near symmetry and orthogonal analysis , demonstrates the best performance.
(a)
(b)
(d)
(b)
5 Results A closer look is given to the models for comparison. The sizes of two input boundary zones are 20 by 40 pixels . Results are given in Figure 5. It could be observed that the transition in the overlap region is smoother in (d) and (e) than in (c) and the seam is less visible.
(a)
(c)
(b)
(a) (c)
(d)
(d)
(b)
(e)
Figure 5. Comparison of results of the algorithms. (a) and (b) are two input boundary zones. (c) is the result from original minimum-cost boundary cut algorithm which directly applied in special values. (d) and (e) are the results obtained from Model I and Model II respectively.
Some other texture synthesis results are given in Figure 6. The original texture, synthesized texture using original minimum-cost boundary cut algorithm, the results from Model I and Model II. In these results, sizes of patches are 64×64 and boundary zone widths are 32.
(c)
(d)
Figure 6. Results comparison of the algorithms. (a) is the sample texture (b) is the result from original minimum-cost boundary cut algorithm which directly applied in special values. (c) and (d) are the results obtained from Model I and Model II, respectively.
270
6 Conclusion
SIGGRAPH93, Computer Graphics Proceeding, Annual Conference Series, Aug. 1993, 221-230.
Two novel models are proposed to compensate for the quality defects caused by the intrinsic properties of patch-based texture synthesis. Experimental results show that by manipulating mult i-resolutions on the different sub-band of wavelet coefficients, seams are made less visible and the transition bands are more natural. Further research is planned to apply these techniques in other image processing and texture related areas.
[13] J. Portilla and E. Simoncelli, A parametric texture model based on joint statistics of complex wavelet coefficients. International Journal of computer vision, Oct. 2000, 40(1):49-70.
References
[14] Z. Bar-Joseph, R. El-Yaniv, D. Lischinski, and M. Werman, Texture mixing and texture movie synthesis using statistical learning. IEEE Transactions on Visualization and Computer Graphics, 2001, 7(2):120-135.
[1] E.E. Catmull, A Subdivision Algorithm for Computer Display of Curved Surfaces. Ph.D. thesis, University of Utah, Dec. 1974.
[15] S. Arivazhagan and L. Ganesan, Texture classification using wavelet transform. Pattern Recognition Letters, Jun. 2003, 24(9-10): 1513-1521.
[2] A. Hertzmann, Algorithms for Rendering in Artistic Styles. Ph. D. thesis, New York University, 2001.
[16] S-J Huang, C-T Hsieh, Coiflet Wavelet transform applied to inspect power system disturbance-generated signals. IEEE transaction on aerospace and electronics system, Jan. 2000, vol. 38, no.1.
[3] L.Y.Wei, Texture Synthesis by Fixed Neighbourhood Searching. Ph.D. Dissertation, Stanford University, 2001. [4] A.A. Efros, and W.T. Freeman, Image quilting for texture synthesis and transfer. SIGGRAPH, 2001, 1-6. [5] V. Kwatra, I.Essa, A. Schodl, G. Turk, A. Bobick, Graphcut Textures: Image and video synthesis using graph cuts, SIGGRAPH, 2003. [6] A. Neubeck, A. Zalesny, and L. Van Gool, Cut-primed smart copying. Texture 2003 Workshop in conjunction with ICCV, 2003. [7] L. Liang, C. Liu, Y. –Q. Xu, b. Guo, and H. -Y. Shum., Real-time texture synthesis by patch-based sampling. ACM Transactions on Graphics, Jul. 2001, 20(3): 127-150. [8] E. H. Adelson, P. J. Burt, A multiresolution spline with application to image mosaics. ACM Transaction. 1983, Graph. 2(4), 217-236. [9] J.Singh, K. J. Dana, Clustering and blending for texture synthesis . Pattern Recognition Letter 25, 2004 619-629. [10] A. Levin, A. Zomet, S. Peleg, and Y, Weiss, Seamless Image Stitching in the Gradient Domain. Proc. of the European Conference on Computer Vision (ECCV), Prague, May 2004. [11] I. Drori, D. Lishchinski, Fast multi-resolution image operations in the wavelet domain. IEEE transactions on visualization and computer graphics. 2003, 395-411. [12] S. J. Gortler, P. Schroder, M. F. Cohen, and P. Hanrahan. Wavelet radiosity. Proceedings of
271
