Efficient Automatic Image Annotation Using Optimized ... - Science Direct

Available online at www.sciencedirect.com

ScienceDirect Procedia Computer Science 58 (2015) 731 – 739

Second International Symposium on Computer Vision and the Internet (VisionNet’15)

Efficient Automatic Image Annotation using Optimized weighted Complementary Feature Fusion using Genetic Algorithm Ajimi Ameer ,SreeKumar.K a

b

a

College Of Engineering Poonjar,Kottayam 686582,India College Of Engineering Poonjar,Kottayam 686582,India

b

Abstract Retrieval of images of user’s interest from a large database is complex since the image content is defined on different features as color,texture,shape.And we can combine these features to give a single feature vector that represents an image. In conventional methods, equal weights are taken for each feature and this will increase the feature dimensionality and certain image features will override the feature that the user is really interested on a particular image. So, to overcome these problems, in this paper, weights of different features should be assigned appropriately using Genetic algorithm(GA) similar to human perception which gives an optimized feature vector of each image. 2015Published The Authors. Published Elsevier B.V. ©©2015 by Elsevier B.V. by This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/). Peer-review under responsibility of organizing committee of the Second International Symposium on Computer Vision and the Peer-review under responsibility of organizing committee of the Second International Symposium on Computer Vision and the Internet Internet (VisionNet’15). (VisionNet’15) Keywords: Genetic Algorithm(GA);Feature vector weights ;Feature fusion,;HOG;SURF;HSV.

* Corresponding author. Tel.: +91-784-585-0403. 㻌 E-mail address: [email protected]

1877-0509 © 2015 Published by Elsevier B.V. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/). Peer-review under responsibility of organizing committee of the Second International Symposium on Computer Vision and the Internet (VisionNet’15) doi:10.1016/j.procs.2015.08.094

732

Ajimi Ameer and K. SreeKumar / Procedia Computer Science 58 (2015) 731 – 739

1. Introduction Automatic image annotation which is also known as linguistic indexing or automatic image tagging is the method of assigning appropriate keywords or captions to an image in multi-class image classification. Images retrieved using trained set of images.In this paper,three phases Genetic algorithm is used to get the weights of different features during training phase. In this method, image representation scheme is achieved using three descriptors, namely, Hue Saturation Value,HSV Color Histogram (10 dimensional vector), Histogram of Oriented Gradients,HOG (81dimensional vector) and Speeded-Up Robust Features,SURF (64 dimensional vector). For each class of images the weights of these features are obtained using genetic algorithm . 2. Related Works Many techniques and approaches were proposed for optimization in automatically annotating images in various fields. For monitoring water quality, Ni Bin Chang et.al. proposed an Integrated Data Fusion and Machine-learning (IDFM) which is an early warning system for the prediction of microcystin concentrations and distribution by measuring the surface reflectance of the water body using satellite sensors[14]. Finally,an image is obtained by the fusion of the temporal, spectral, and spatial properties of two or more images. For improved content based retrieval a genetic algorithm based neural network optimization is proposed in [20]. Features extracted from the image using Discrete Cosine Transform, relevant features are then extracted using the method of information gain and for classification Gaussian based Fuzzy Feed Forward artificial Neural is used and the learning and momentum rate optimized using the neuroevolution via Genetic algorithm. Joey Mark Diaz et.al. used genetic algorithm for feature (genes) selection for the support vector machine (SVM)and artificial neural network in Lung Cancer Classification research in order to classify lung cancer status of a patient . Genetic algorithm (GA) successfully identified genes that classify patient lung cancer status with notable predictive Performance[16].Content-based image retrieval (CBIR) of medical images, according to its domain specific image features, is a valuable tool for physicians[17]. The method has three major steps: 1. Features extracted from the CT images using Coif let wavelets; 2. The features extracted are classified using Support Vector Machine; 3. The parameters of the SVM are optimized using Particle Swarm Optimization (PSO), and modified PSO with a genetic algorithm. Ji Zhoung et.al proposed a Land Surface temperature retrieval from Moderate-resolution Imaging Spectroradiometer (MODIS) data by regression models integration and using GA in some arid regions[22]. Zhao and Song applied the Genetic algorithm in MODIS data combined with linear spectral mixing model [19] to retrieve component temperature.Ashok Samal et.al. proposed a method by integrating measures for searching satellite imagery[21].In geospatial imagery retrieval, they used a feature selection based structure learning algorithm. Then the parameters of the structure models are optimized through GA. The system is used to classify different land zones in Greece with the help of IKONOS imagery. 3. Proposed Work The proposed system works in three phases: Training phase ,Fusion Phase and Annotation phase.Training phase proceeds in two steps. First the Histogram Of Oriented Gradients(HOG),Speed-Up Robust Features(SURF),and Hue Saturation Value(HSV) color features are obtained from training set of images. Secondly, average feature descriptor representing each class is obtained.In fusing phase ,weights of different features are found out using Genetic algorithm for each class of images.In the annotation phase first the features of each image is extracted and synthesized as in training phase,then these images are classified according to the similarity measure and finally these images have been annotated based on the model of each class obtained in the training phase and the weights of features obtained during the fusing phase.


3.1. Training Phase Step 1: Feature Extraction Feature extraction is the most important stage of any classification and image annotation..For each class, hundred images are trained. x x x

SURF Feature Extraction: The proposed system uses speeded up robust feature extraction method (SURF) in order to get the features of images. Using SURF, we will get the information not obtained from edge detector or corner detector, that is ridge detection and it is rotation and scale invariant transformation. Color Feature Extraction: Hue Saturation Value (HSV) color model is used for color features extraction. HSV is used to get different perception of colors. A 10 dimensional feature vector is used to describe HSV values of each image in training and testing phase. Texture Feature Extraction: Histogram of Oriented gradients(HOG) model is used to extract shape and texture features from the images.It is suitable for object and human detection.This is done through creation of cell histograms and orientation binning. It is done through giving each pixel in a cell a weighted vote based on its orientation.

Step 2: K-Means Clustering Now N number of rows are used to represent N number of training images. N feature vectors are clustered in to k-clusters using Kmeans clustering. Kmeans clustering is a method of clustering to partition n images into k clusters in which each image belongs to the cluster with the nearest mean.It is a simplest unsupervised learning algorithm. 3.2. Fusion Phase x

Genetic algorithm for optimized feature weights Genetic algorithm mimics the process of natural selections.Initially a population of N chromosomes are created which has the same length of the feature vector.From the initial population best chromoseomes are selected for next generation. And as a result of finding the appropriate weights using GA feature dimensionality is reduced so that relevant features are preserved ignoring remaining.Genetic algorithm can solve problems which has multiple solutions by clearly defining the fitness function.

x

Basic Genetic algorithm operations

Genetic algorithm is a heuristic technique which involves stochastic optimization.Algorithm consists of following phases: selection, crossover and mutation. The basic genetic algorithm is as follows: Step 1: [start] Start with initial population of N chromosomes specific to the solution domain.GA begins its search from a random population. Step 2: [Fitness] A fitness function f(x) is created for evaluating the solution space. Step 3: [New population] Next generation of population is iteratively created by repeating the following steps until the termination condition satisfied. Step 4: [Selection] Reproduction or Selection is the first operator that should be applied in the GA by selecting the best offsprings from the current population and then they inserted into a mating pool. Step 5: [crossover] It is applied to avoid getting trapped on a locl minima by crossovering or exchanging a part of the selected parents with a probability to give a new offpring that resembles the parents. Step 6: [Mutation] Mutation alters the offspring hopefully to give a better offspring. Step 7: [Accepting] Newly generated offspring is placed in the mating pool. Step 8: [Replace] If better offsprings are not generated, they are replaced and best-so-far offsprings selected for next generation. Step 9: [Test] If the termination condition is satisfied ,then the top best solution is returned. Step 10: [Loop] Go to step 3 for better fitness evaluation in the next iteration.

733

734


Fig. 1. Genetic Algorithm Overview

Genetic algorithm is used for finding fittest solutions exploring through large solution domain that may not find in a lifetime. GAs run on set of solution points at a time rather than performing one at a time,so the problem of being stucked on a local minima is reduced. By coding of variables,GAs can discretizes the solution space even though the function is continuous.Different processes in fusion phase is discussed after the annotation phase. 3.3. Annotation Phase Different feature extraction process in annotation phase is similar to the feature extraction process used in the training phase. Step 1: Feature Extraction Feature extraction is the most important stage of any classification and image annotation..For each class, hundred images are trained. x x x

SURF Feature Extraction: The proposed architecture uses speeded up robust feature extraction method (SURF) to extract the features of both testing and training images. Color Feature Extraction: Hue Saturation Value (HSV)color model is used for color feature extraction,. HOG Feature Extraction: HOG feature operates on localized cells,therefore it is invariant to geometric transformation. It is suited for human detection in images.

Step 2: Feature Fusion Image features should be fused accordingly in order to retrieve the correct image from the dataset as desired by the user. In order to get the weights of different features appropriately similar to human perception, Genetic algorithm is used.Hence, instead of assigning equal weights to different features,

735


variable ranges of weights are assigned using GA so that image feature that user is actually focusing on an image gets higher weightage. Step 3: Annotating test images Since the representations and actual meaning of each image feature is different, their value ranges are dissimilar, hence ,similarity scores of different features of different images cannot be compared meaningfully. So,to overcome this problem of value range in different features, feature vectors should be normalized before finding the weights. Suppose Q is the given query image. By calculating the eucleidan distances between all images in the database and the given query image Q, we will get, similarity score set {Si}, where i = 1,2,3.…N, and N means the total number of images in database. Thus, normalization of the similarity score can be achieved as: SNi = max {Si} - min {Si}

(1)

Si-min {Si} In this work the weights are obtained from genetic algorithm in the fusion phase by comparing the similarity measure of some known sample images of each class and the averaged feature vectors of each class and then taking the best sample image that gave the best score on comparison. Then the similarity measure of these images with the averaged feature vector of a class are given as input to the fitness function of genetic algorithm. The multi-feature similarity scores of known images and each of the 10 classes are calculated using the following equation: Sim(C,K)= √[ ∑(Csurf - Ksurf)2 + ∑(Chog - Khog)2 + ∑(Chsv - Khsv)2 ]

(2)

where C is the average feature vector representing each class and K is the feature vector of known images of each class. Csurf is the average feature vector of SURF feature representing each class and Ksurf is the SURF feature

736


Fig. 2. Proposed Architecture

737


vector of known images of each class. Chog is the average feature vector of HOG feature representing each class and Khog is the HOG feature vector of known images of each class. Chsv is the average feature vector of HSV feature representing each class and Khsv is the HSV feature vector of known images of each class. In this work, five known images for each class And weights of features find out using the fitness function as: SFi = SNsurf(i). Wsurf + SNhog(i). Whog + SNhsv(i). Whsv

(3)

Wsurf + Whog + Whsv

SFi -fused similarity score, SNsurf(i)- normalized surf feature similarity score, SNhog(i)- normalized hog feature similarity score, , SNhsv(i)- normalized hsv feature similarity score, Wsurf -Weight of surf feature similarity score,Whog-Weight of hog feature similarity score, Whsv -Weight of hsv feature similarity score. In this work, Wsurf+ Whog +Whsv=1

(4)

So, the weights Wsurf, Whog, Whsv can be obtained through GA. Finally, in the annotation phase, feature vector of each query image,Q is multiplied with the each of the feature weight obtained through GA before using it for recognition as: Sim(Q,C)= √[ ∑(Qsurf - Csurf)2]. Wsurf + √[∑(Qhog - Chog)2]. Whog + √[∑(Qhsv - Chsv)2 ] . Whsv

(5)

Where C is the average feature vector representing each class, Qsurf, Qhog, Qhsv are the feature vectors of SURF,HOG and HSV features respectively representing each query image Q and Csurf, Chog, Chsv. is the average feature vector of SURF,HOG and HSV features respectively representing each class of images.Finally,the images which has higher similarity with the query image are retrieved. 4. Experimental Results The proposed system was implemented and experiments were conducted with standard Corel 1000 dataset . The Corel 1000 dataset contains 1000 images spread across 10 classes. 1000 images, 100 from each class were used in training phase and annotation was performed on the entire dataset of 1000 images. Euclidian distance was used for distance similarity measurement in the annotation and classification phase. The proposed system was evaluated for performance using statistical approaches by computing parameters like precision, recall, F-score and accuracy. PRECISION

RECALL

1 Without GA With GA

0.9

0.8

0.8

0.7

0.7

0.6

0.6 Recall

Precision

0.9

1

0.5

0.5

0.4

0.4

0.3

0.3

0.2

0.2

0.1

0.1

0 African Beach Building

Bus

Dinosaur Elephant Flower

Horse

Fig..3.Precision Estimation with and without GA

Meal Mountain

With GA Without GA


Bus

Dinosaur Elephant Flower

Horse

Fig.4.Recall Estimation with and without GA

Meal Mountain

738


ACCURACY

F-SCORE 1

1 With GA Without GA

0.9

0.8

0.8

0.7

0.7

0.6

0.6 Accuracy

F-Score

0.9

0.5

0.5

0.4

0.4

0.3

0.3

0.2

0.2

0.1

0.1


Bus Dinosaur Elephant Flower

Horse

Fig. 5. F-Score Estimation with and without GA

Meal Mountain

With GA Without GA


Bus Dinosaur Elephant Flower

Horse

Meal Mountain

Fig.6.Accuracy Estimation with and without GA

Fig. 7. Images automatically annotated as “elephant” with weighted feature fusion using GA but annotated as “horse” with equal weights without GA.

Crossover probability in this work is 0.8 and mutation as mutationadaptfeasible since constraints on sum of weights of features are given. Since GA is an optimization algorithm it does not gives the same result in all iterations,the results will be slightly different in each iteration.When the number of iterations increases in genetic algorithm,weights obtained becomes much more accurate and yields better result. 5. Conclusion The current work aims at automatic image annotation using the optimized weights obtained through the Genetic algorithm.As weights of different descriptor features are assigned appropriately,certain image features in the query image where traditional algorithms had failed due to multiple local optimas and where objective functions is not smooth becomes more accurate.There is a lot of scope for improving the proposed system including making the Genetic algorithm much more interactive to the user by giving the user freedom for selecting the appropriate images


for next generation and also by additionally improving the constraints for obtaining the weights and selecting probability of selection and crossover more accurately. References 1. Chandrashekhar G.Patil, Dr.Mahesh T.Kolte2, Dr.Devendra S.Chaudhari, Fusion at Features Level in CBIR System using Genetic Algorithm,2013 2. Sapthagiri.k, Manickam , An Efficient Image Retrieval Based on Color, Texture (GLCM & CCM) ,features, and Genetic- Algorithm,2013 3. K. Aparna, Retrieval of Digital Images Based On Multi-Feature Similarity Using Genetic Algorithm, 2013 4. Ruaa Mohammed Hamza, Dr. Tawfiq A. Al-Assadi, Genetic Algorithm to find optimal GLCM features,2012 5. Shafimirza,Dr.J.Apparao, Retrieval Of Digital Images Using Texture Feature With Advanced Genetic Algorithm,2012 6. S.Sreenivas Rao , Mr. K.Ravi Kumar, Dr. G. Lavanya Devi, Texture Based Image retrieval using Human interactive Genetic Algorithm,2013 7. S.Gopalakrishnan, Dr. P. Aruna, Retrieval of images based on low level features using genetic algorithm,2014 8. K.Kalaiyarasi ,A.Kabilar M. Image Retreival Based On Colour, Texture and Shape Analysis Using Genetic Algorithm,2014 9. P.Kishore Kumar, M.Radhika, Using Genetic Algorithm Image Retrieval Based on Multi- Feature Similarity Score Fusion,2014. 10. Dr.Mahesh.T.Kolte, Chandrashekhar,Improvement in Performance of CBIR by using fusion and Evolutionary Computation,2014 11. Anita Nanasaheb Ligade, Manisha R. Patil, Optimized content based image retrieval using genetic algorithm with relevance feedback technique,2013 12. Miguel Arevalillo-Herraez, Francesc J. Ferri, Salvador Moreno-Picot, An interactive evolutionary approach for content based image retrieval,2013 13. M. Venkat Dass, Mohammed Rahmath Ali, Mohammed Mahmood Ali, Image Retrieval Using Interactive Genetic Algorithm,2014 14. Ni-Bin Chang and Benjamin Vannah, Compartive Data Fusion between Genetic Programing and Nueral Network Models for Remote Sensing Images of Water QualityMonitoring, 2014 15. Itedal Sabri Hashim Bahia, Using Artificial Neural Network Modeling in Forecasting Revenue: Case Study in National Insurance Company/Iraq, 2013 16. Joey Mark Diaz, Raymond Christopher Pinon, Geoffrey Solano, Lung Cancer Classification Using Genetic Algorithm to Optimize Prediction Models,2013 17. Ms.N.T.Renukadevi and Dr.P.Thangaraj , Performance analysis of optimization techniques for medical image retrieval,2014 18. Miguel Arevalillo-Herr´aez, Francesc J. Ferri, Salvador Moreno-Picot, An interactive evolutionary approach for content based image retrieval,2013 19. Song, X.N.; Zhao, Y.S. Study on component temperatures inversion using satellite remotely sensed data. Int. J. Remote Sens. 2007. 20. C.Ramesh babu durai, V.Duraisamy, C.Vinothkumar,Improved Content Based Image Retrieval Using Neural Network Optimization with Genetic Algorithm ,2012. 21. AshokSamala,,SanjivBhatiab,PrasanthVadlamania, DavidMarxc, Searching satellite imagery with integrated measures,2009 22. Ji Zhou , Xu Zhang , Wenfeng Zhan and Huailan Zhang ,Land Surface Temperature Retrieval from MODIS Data by Integrating Regression Models and the Genetic Algorithm in an Arid Region ,2014. 23. Dengsheng Zhang, Aylwin Wong, Maria Indrawan, Guojun Lu, Content-based Image Retrieval Using Gabor Texture Features. 24. Chandrika L, Implementation Image Retrieval and Classification with SURF Technique,2014 25. MS. R.Janani,Sebhakumar.P ,An Improved CBIR Method Using Color and Texture Properties with Relevance Feedback,2014. 26. Nadia Baaziz, Omar Abahmane and Rokia Missaoui, Texture feature extraction in the spatial-frequency domain for content-based image retrieval

739