Study on Feature Selection and Machine Learning ... - IEEE Xplore

2014 International Conference on Information Technology and Multimedia (ICIMU), November 18 – 20, 2014, Putrajaya, Malaysia

Study on Feature Selection and Machine Learning Algorithms For Malay Sentiment Classification Ahmed Alsaffar Center for AI Technology, FTSM University Kebangsaan Malaysia,UKM 43000 Bangi Selangor, Malaysia [email protected]

Nazlia Omar Center for AI Technology, FTSM University Kebangsaan Malaysia,UKM 43000 Bangi Selangor, Malaysia [email protected] classifiers [2–4]. The semantic orientation (SO) approaches utilize lexical resources, such as SentiWordNet, to measure the polarity orientation of the words to classify reviews [4, 5]. Both approaches have advantages and drawbacks. For example, a high-quality supervised machine learning approach critically depends on the availability of large training corpora that are often impossible or difficult to find in every language and every domain. In addition, the SO approaches require a large amount of linguistic resources, the availability of which typically depends on the language [3]. Over the last few decades, most efforts exerted to solve sentiment analysis concern the English language, for which a large amount of resources and tools for natural language processing, especially for sentiment analysis, are available. The worldwide interest on sentiment analysis can be attributed to the fact that the sentiments and feelings expressed in social media are often in the language of the users. Consequently, the need for more studies on sentiment analysis and for construction of resources and tools for subjectivity and sentiment analysis in more new languages, such as Malay, is growing. Malay language used as the formal language in four countries (Malaysia, Indonesia, and Brunei, and Singapore) There are about 215 million people in various cities of these countries considered as native of Malay language..Malay is underresourced in terms of language technologies and lexical resources. Thus, we believe that focusing on Malay opinion mining and sentiment analysis would be lucrative. The main motivation of this work is to develop sentiment classification systems in languages other than English. This study specifically focuses on Malay sentiment classification. However, Most of studies focus on analyzing the users’ opinions based on English language. There has been a very limited amount of research that focuses on sentiment analysis in the Malay language. This paper design several sentiment classification models for Malay sentiment analysis using 7 methods of feature selection; “information gain” (IG), “principal components analysis” (PCA), “Relief-F”, “Gini Index”, “uncertainty”, “Chi-square”, and “support vector machines” supporting 3 machine learning approaches; SVM, (SVM) “Naive Bayes” (NB), and “Knearest neighbor” (KNN To select the appropriate methods for the automatic sentiment classification in the online Malaywritten reviews effectively, this work has three main objectives. The first aim is to evaluate the most accurate

Abstract—Online social media is used to show the sentiments of different individuals about various subjects. Sentiment analysis or opinion mining has recently been considered as one of the highly dynamic research fields in natural language processing, web mining, and machine learning. There has been a very limited amount of research that focuses on sentiment analysis in the Malay language. This study investigates how feature selection methods contribute to the improvement of Malay sentiment classification performance. Three supervised machine-learning classifiers and seven feature selection methods are used to conduct a series of experiments for the effective selection of the appropriate methods for the automatic sentiment classification of online Malay-written reviews. Findings show that the classifications of Malay sentiment improve using feature selections approaches. This work demonstrates that all feature reduction methods generally improve classifier performance. Support Vector Machine (SVM) approach provide the highest accuracy performance of features selection in order to classify Malay sentiment comparing with other classifications approaches such as PCA and CHI square. SVM records 87% as experimental accuracy result of feature selection. Keywords—Feature Selection, Sentiment analysis, Machine Learning, NLP, Classifications.

I.

INTRODUCTION

Sentiment analysis is a highly active area of research that involves the computational study of opinions, evaluations, and reviews about products, services, and policies that are expressed in the written language, as well as the construction of sentiment corpora and dictionaries [1]. The Internet is moving toward Web 2.0. There are many social communication application allow the organizations and individuals to gather the social contents online which known as social media network The social network users increase rapidly due to free and interactive online communication service. Thus, the users’ can share their opinions and feelings effectively. The mining opinions of social communication are difficult to analyzed due opinions dimensions complexity i.e. need of deep understanding of explicit and implicit features, regular and irregular, and syntactical and semantic rules of the proper use of a language. Thus, the supervised and unsupervised learning methods could be applied to address this issue. . In supervised approaches, sentiment corpora are used to train

978-1-4799-5423-0/14/$31.00 ©2014 IEEE

270


preprocessing consists of the method of cleaning the raw data in order to transform the noisy data into clean ones. Two important techniques that are performed in sentiment mining are tokenization stop word removal [24, 25]. The tokenization is the processes that segment the texts into sentences and words. In the English language, words are bounded by whitespace and optionally preceded and followed by parentheses, quotes, or punctuation marks. Therefore, the tokenization divides the character sequence based on the whitespace positions or other punctuation marks between words in the sentence. In addition, it cuts off the parentheses and punctuation marks to obtain the sequence of tokens. On the other side, stop word removal is the process of removing words that have high frequency which are not important to the sentiment of the sentence. Words such as; ‘a', ‘the', ‘or' are likely to be considered as stop words which have been listed in

feature selection method of Malay sentiment classification. Second, this work aims to determine which learning method performs best. Third, this research also investigates how feature selection methods contribute to the improvement of the classification performance of three machine-learning classifiers on Malay sentiment analysis. All models are evaluated on the basis of an opinion corpus for Malay (OCM), which was collected from a variety of Web pages about reviews in the Malay language. II.

RELATED WORK

A. Sentiment Analysis Sentiment analysis is studied extensively by many researchers in the last few years. Research literature include some excellent and comprehensive reviews that are related to subjectivity and sentiment analysis [6–10], and describe the different techniques used for sentiment analysis in text documents. Sentiment analysis works focus on determining the sentiment polarity at one of the three levels: the document level [11, 12], the sentence level [13–15], or the feature level [16]. In the document level sentiment analysis, the entire review is considered. The polarity is determined based on the overall sentiment of the entire review. In this case, the opinion holder is usually assumed to be a single person or source. For the document level sentiment analysis, several machine learning techniques are employed. Traditional classification methods such as SVMs, NB, decision tree, and KNN [2, 3, 7, 11, 17] are applied to the document level sentiment classification. However, the success of this classification depends on several factors, such as the topic, domain, and timely period represented by the training data. Furthermore, the SO approach or unsupervised learning method is used for sentiment classification. The SO approach does not require any prior training. Instead, this approach uses lexical resources to calculate how far a word is positively or negatively inclined. Kamps et al. [18] use lexical relations in the sentiment classification. Esuli and Sebastiani [19, 20] propose a semi-supervised learning method with WordNet as the lexical resource. This method proposed by Esuli and Sebastiani [19, 20] starts by expanding an initial seed set from WordNet, where the basic assumption is that terms that have similar orientations tend to have similar sentiments. Finally, transfer learning approaches are also employed, particularly when there are no labels on the data [21–23]. The transferbased approaches aim to utilize data from other domains or time periods to help the current learning task.

C. Sentiment Classification Using Machine Learning There are few related works that are conducted for sentiment analysis and classification models in Malay. The strengths and weaknesses of each model are also discussed in the next sections. Samsudin et al. [24] propose a model that uses a method, where noisy texts that are available in Mixed Malay Language texts are normalized with the creation of the Malay Mixed Text Normalization Approach. Puteh [25] studies preprocessing methods for stemming Malay text, with the use of the Reverse Porter Algorithm and Backward Forward Algorithm. Additionally, Puteh [25] also uses the artificial immune network to retrieve opinions from newspaper articles in Malay. Accordingly, the performances of the different types of traditional machine learning technique classifiers, like SVM, NB, and K-NN, must be analyzed and compared when used with the different feature selection methods (FSMs). In addition, the FSMs that are not yet applied to any Malay text classification must be investigated, including the effect of these FSMs on the performance of the state-of-the-art FSM machine learning algorithms.

III.

METHODOLOGY

Figure I shows the methodology used in our Malay sentiment analysis system. First, the incomplete and inconsistent data are eliminated through pre-processing tasks. The data must be pre-processed to perform any further data mining functionality. Next, FSMs are applied to identify the discriminating terms for training and classification. Finally, a supervised machine learning classifier is used to classify the sentiments into positive and negative classes.

B. Preprocessing

A. Pre-processing In any user-generated data, information is usually mixed with noisy data, which makes the analysis more difficult. The data are pre-processed to deal with misspellings, abbreviations, or erroneous punctuation marks. In our system, all the Malay reviews underwent the three pre-processing steps:

Preprocessing is a process to perform a preliminary processing on raw data to prepare it for another processing procedure. It is commonly used as a preliminary sentiment mining practice, data preprocessing transforms the data into a format that will be easily and effectively processed computationally. Data

271

2014 International Conference on Informa ation Technology and Multimedia (ICIMU), November 18 – 20, 2014, Putrajaya, Malaysia

2) (PCA): This method geneerate new variables depend on linear relation between origiinal variables. . Thus, the main aim PCA is to generate new n set of solutions based on current data to support the Varity of solutions based on specific rules. Let x be the original D-dimennsional observation vector that represents a review and let dimensional d observation vector representing a review and be a new multi-dimensional extracted vector obtained byy a linear transformation: .The coefficients of the D ×d matrix A can be found as follows. Let ∑ be the covariannce matrix of x. This matrix has D eigenvalues: 0. It can be deduced that matrix A is composed of D eigeenvectors that correspond to the d largest eigenvalues , .1 Every eigenvector is a column of the matrix A.

1) Tokenizing of the words depennding on the white space and punctuation marks. 2) Stop word removal task: elimiination of intrinsic words, non-words, punctuationn marks, and stop words. 3) Spelling correction: a Malaay user-generated content on social media contaiins a high number of abbreviations, such as sbb, bg, and tgk ().Thus, m is applied in this a spelling correction algorithm step. B. Feature Selection Methods FSM is one of the most important tasks that will enhance the performance of the sentiment classificatioon system, because the FSM will select the most predictive feaatures. In addition, the FSM reduces the high dimensionalityy of the data and removes irrelevant, redundant, and noisy datta. Thus, the FSM can help in building faster, cost effectiive, and accurate sentiment classification models. In this workk, the selection of the FSM algorithms is influenced by thee data size, data consistency, and the need to investigate the t most efficient FSMs. In this section, we briefly introducce seven effective FSMs: IG, PCA, Relief, Gini Index, uncertaainty, Chi-squared, and SVM-based methods. These seven meethods compute a score for each individual feature, and then select s a predefined size for the feature set.

3) (SVM): this approach focuuses on classifying the features depend on its weights. The feaatures weights calculated based on the features importance. The T features that lay to vector distance are important whilee the other features are not important. Suppose we have vectors v x1….xi that have set of features; the important featurres are the features that have higher weights (WF). Thus, thhe other features that have low weights could be reduced. 4) Relief-f: this method work on classify the features of same class based on their relations with each other. Therefore, the weak related features will be dismissed. . the features selected randomly from the traaining data. The highest weights of features will save in the sam me class; the dismissed features or low weights may assign to other classes if the weights of these features are high in other proposed classes. However, the same feature may be included in i different classes. Specifically, to allocate the value as the weight for each term feature f, the relief feature attempts to find the t best estimate of from the following probabilities: Wx =

diff .Value of f − neaarest instances from diff .class diff .Value of f − nea rest instances from same c l a ss

5) Chi-square: This method used widely in the features selection implementations. The T main aim of this method is to calculate the relation betw ween the features and its class; the classes need to have sttrong relation with its features (Galavotti, Sebastiani, and Simi, S 2000). The formulation of this method as follows: , Fig. 1. The Malay Sentiment Analysis Methodology

χ

1) (IG): This method considered as effeective solution of features selection. IG measures the given values v of features based on the classes attributes rules of featuures classification. The best classification of features is the low west size of feature vectors. Thus, the vectors features could be reeduced efficiently.

t

m max

,

where A is t and c appearannce number together and, B is t appearance number without c. C is c appearance number

272


without t. , D is the number of the cases that c and t not appear together, and N is the total number of documents. 6) Gini Index: this method introduced by Wenqian Shang et al. (Wenqian et al., 2006), depend on the theory of Gini Indexing. The original form of this algorithm is used to measure the impurity of attributes toward classification and to find the best split of attributes in decision trees. The quality of the attribute improves with the decreasing impurity size.

ensure the efficiency of relation between the proposed classes and the features that belong to these classes. To classify the most probable class c* for a new document d, NB computes through the following equation: c

argmax

|

The NB classifier calculates the posterior probability as follows:

| |

Gini

|

|

I. In this formula, if feature t appears in every document of class , then the maximum value Gini 1 is obtained.

3) K-nearest neighbor classifier: The KNN is a well-known example-based classifier. The KNN is called lazy learners because the decision of this classifier on how to generalize beyond the training data until each new query instance is encountered, is deferred. To categorize a review, the KNN classifier ranks the neighbors of the particular review among the training reviews. Subsequently, the KNN uses the class labels of the K most similar neighbors.

7) Uncertainty-based term selection: This operator calculates the relevance of the attributes of the given training set by measuring the symmetrical uncertainty with respect to the class. The relevance is calculated through the following formula: 2 | Relevance

Given a test review d, the system finds the K-nearest neighbors among the training reviews. The similarity score of each nearest neighbor review to the test review is used as the weight of the classes of the neighbor review. The weighted sum in the KNN classification can be written as follows:

C. Classification Methods In this study, three classifier methods are used in Arabic sentiment classification, namely, the NB, SVM, and KNN methods, which are used because of the simplicity, effectiveness, and accuracy of these methods. The following are the brief descriptions of these methods.

score d, t

KNN

sim d, d δ d , c

where KNN(d) indicates the set of K nearest neighbours of review d. If d belongs to c , then δ d , c equals one; otherwise, it is zero. For test reviewd, it should belong to the class that has the highest resulting weighted sum.

1) SVM Classifier: This approach introduced by Cortes and Vapnik (1995). SVMs are very popular techniques for text categorization used in machine learning implementations. SVMs are considered as one of the most effective classification methods, according to the performance of these techniques on text classification. SVM minimize the complexity of original and large data set through break the proposed dataset into many classes before decide the effected vectors that belong to each class. Thus the decision making of feature selection will be simplified. The optimization procedure of SVMs (dual form) is to minimize the following:

IV.

EXPERIMENTAL SETTING

This work conducts several experiments to evaluate our model. First, we evaluate the performance of the classification algorithms. The performance of these classification models are measured based on an OCM. The corpus contains 2,000 reviews collected from online Malay social media and blogs; 1,000 of these reviews are considered positive, while the other 1,000 are considered negative. All of the algorithms are evaluated through K-fold crossvalidation. This work aims to analyze the best method of r features reduction Malay sentiment. To evaluate the accuracy performance of proposed methods of machine learning many procedures are followed; (1) assign the correct reviews of the category are calculated (“True Positive” (TP)) , (2) assign the incorrect reviews are calculated (“False Positive” (FP)) (3) assign the incorrect reviews that not calculated (“False Negative” (FN)), and (4) assign the correct reviews that not calculated (“True Negative” (TN)). also, F1 and Macro-F1 are measured. The following describes these metrics:

, :

∑

0; 0

2) Naive Bayes: this approach manage the features as table of vectors as central pool of features; the features classified to the proposed classes based on specific classification rules to

273


Precision Recall F_1

TP/ TP TP/ TP

FN

2 Recall Precision / Recall F

V.

addition, we look at the effects of the feature set size on the classification performance of the NB classifier. Most of the time, the highest performance is achieved when the feature size is 200. The macro-averaging F-measure results of the SVM classifier with each of the seven FSMs are shown in Table III. The results of the SVM classifier with five of the seven FSMs significantly outperform the original result of the SVM classifier. The best performance of the SVM is achieved when the SVM-based and Gini index FSMs are used (see the average row in Table III). According to Table III, the best performance (87%) of the SVM classifier is achieved when the SVM-based method FSM is used and the feature size is 500. The macro-averaging F-measure results of the KNN classifier with each of the seven FSMs are shown in Table IV. Six of the seven FSMs significantly outperform the original classifier. The best performance of the KNN classifier is achieved when the SVM-based and IG FSMs are used (see the average row in Table IV). According to Table IV, the best performance (75.15) of the KNN classifier is achieved when the GI FSM method is used and the feature size is 200.

FP

Precision

1 m

EXPERIMENTAL RESULTS

First, we start by examining the overall performance of each one of the three classifiers on the Malay sentiment analysis without feature reduction. The three classifiers (NB, SVM, and KNN) are initially applied to the entire reviews-term feature space. Table I shows a summary of the experimental results using the NB, SVM, and KNN classifiers. As shown in Table I, the highest performance on Malay sentiment classification is obtained with the SVM classifier. However, the worst performance is obtained with the K-NN classifier. SVM minimize the complexity of original and large data set through break the proposed dataset into many classes before decide the effected vectors that belong to each class. Thus, SVM is efficient method for the medium and large set of features. However, KNN ranks the neighbors of the particular review among the training reviews. Subsequently, the KNN uses the class labels of the K most similar neighbors. Thus, KNN face difficulty to classify the large set of features.

TABLE II. MACRO-AVERAGING PRECISION VALUES FOR THE (NB) CLASSIFIER WITH THE SEVEN FSMs Feature size 100 200 300 400 500 Average

TABLE I. PERFORMANCE (THE AVERAGE VALUE OF MACRO-F1 AND THE F-MEASURE FOR EACH CLASS) OF THE NB, SVM, AND KNN CLASSIFIERS Macro F1-MEASURE NB KNN SVM

71.10 58.85 74.50

F1 Measure For the Positive Class 72.13 58.99 75.31

SVM 83.15 85.95 84.65 82.55 80.80 83.42

PCA 62.60 64.90 66.20 66.50 68.25 65.69

IG 84.50 84.95 83.45 81.85 80.35 83.02

CHI 80.55 81.20 81.20 80.60 79.70 80.65

RE 72.60 74.50 73.80 73.35 73.20 73.49

GI 81.05 85.30 83.25 82.05 80.20 82.37

CER 82.90 86.45 84.10 82.30 80.40 83.23

TABLE III. MACRO-AVERAGING PRECISION VALUES FOR THE (SVM) CLASSIFIER WITH THE SEVEN FSMs

F1 Measure For the Negative Class 69.99 58.70 73.62

Second, we study the effects of the different feature selection methods on the classifier performance, to show the behavior of each classifier with the different feature selection methods. In these experiments, each time one FSM from the seven FSMs (IG, PCA, SVM, Relief, Chi, Gini, and uncertainty) is used to select the feature spaces, the effects of the different feature selection methods is studied by varying the size of the top rated features. These features are selected from the feature space at the different sizes of 100, 200, 300, 400, and 500. The macro-averaging F-measure results of the NB classifier with each of the seven FSMs are shown in Table II. The results of the NB classifier with six of the seven FSMs significantly outperform the original result of the NB classifier. The best performance of the NB classifier is achieved when the SVM-based and uncertainty FSMs are used (see the average row in Table II). According to Table II, the highest performance (86.45) of the NB classifier is obtained when the uncertainty method is used and the feature size is 200. In

274

Feature size 100 200 300 400 500 Average

SVM 78.60 83.30 86.05 86.00 87.00 84.19

PCA 59.85 62.20 62.90 63.60 68.45 63.40

IG 76.75 81.70 82.45 82.10 81.90 80.98

CHI 77.20 79.35 80.90 79.80 78.70 79.19

RE 67.20 74.65 74.75 76.00 75.20 73.56

GI 77.60 82.20 82.10 82.40 82.15 81.29

CER 72.25 78.60 81.00 79.65 78.65 78.03

TABLE IV. MACRO-AVERAGING PRECISION VALUES FOR THE (KNN) CLASSIFIER WITH THE SEVEN FSMs Feature size 100 200 300 400 500 Average

SVM 73.00 73.35 74.85 72.10 68.75 72.41

PCA 59.85 62.20 62.90 63.60 68.45 54.04

IG 73.80 73.70 72.15 70.45 70.35 72.09

CHI 68.65 69.10 67.60 66.45 65.30 67.42

RE 65.90 68.75 68.30 66.25 65.85 67.01

GI 73.40 75.15 71.90 71.30 67.90 71.93

CER 69.40 69.85 69.40 66.50 64.65 67.96

By comparing the classifier performances (Tables II, III, and IV), the SVM algorithm evidently outperforms both NB and KNN algorithms. To explain this finding, SVMs use a refined structure that acknowledges the fact that most sentiment analysis features are relevant. SVMs also have the ability to handle large feature spaces. Furthermore, the highest average accuracies are obtained when the feature selection operations


[4]

are done through the SVM-based method. In general, using the FSMs positively contributed to the performance of all the classifiers (Tables I, II, III, and IV). In fact, other FSMs use a simple calculation based on the existence or the lack of the feature in a class, to determine if the feature is relevant or not, whereas the SVM-based method uses a large feature space to classify whether a feature is relevant to a class or not. In the third experiment, we aim to study and compare the performance of the three classifiers in terms of the macroaveraging F-measure with each FSM, using different sizes of feature sets, as shown in (Tables II, III, and IV). The results show that the performances of the three classifiers with one FSM vary when using different numbers of features. In addition, there is no superior classifier for all the feature selection algorithms. As shown in (Tables II, III, and IV), both the size of the features and the FSMs are key factors in determining the appropriate classification method.

[5] [6] [7] [8] [9] [10] [11]

VI. CONCLUSIONS AND FUTURE WORK

[12]

This paper presents an extensive study on the effect of seven FSM methods on three machine learning classifier selections for Malay sentiment classification. The main contribution of this work is to effectively select the appropriate methods for the automatic sentiment classification in the online Malaywritten reviews. This paper attempts to determine which feature selection method performs best for sentiment classification on Malay documents. This paper also investigates how feature selection methods contribute to improving the classification performance of the three machine learning classifiers on Malay sentiment analysis. The results indicate that there is no superior classifier for all the feature selection algorithms, and there is no superior FSM method for all dataset sizes. The results also demonstrate that using the best seven FSMs yields improved results compared with those obtained using the original classifier, particularly with the NB and SVM classifiers. Finally, the results demonstrate that the SVM classifier with the SVM-based FSM achieves the best performance for the Malay sentiment classification, with an accuracy of 87%. SVM provide accurate results for the medium and large set of features more than other features such as NB and KNN. Our future works will focus on developing a Malay Sentilexicon and a Malay large sentiment corpus, investigating the implementation of SO approaches and the implementation of some optimization algorithms, to address the feature selection problem for Malay sentiment classification.

[13]

[14]

[15]

[16]

[17]

[18] [19]

[20] [21]

REFERENCES [1] [2]

[3]

[22]

S. Clematide, S. G. Υ, M. Klenner, S. Petrakis, R. Remus, J. Ruppenhofer, et al., "MLSA – A Multi-layered Reference Corpus for German Sentiment Analysis," 2011. Q. Miao, Q. Li, and D. Zeng, "Mining fine grained opinions by using probabilistic models and domain knowledge," in Web Intelligence and Intelligent Agent Technology (WI-IAT), 2010 IEEE/WIC/ACM International Conference on, 2010, pp. 358-365. M.-T. Martín-Valdivia, E. Martínez-Cámara, J.-M. Perea-Ortega, and L. A. Ureña-López, "Sentiment polarity detection in Spanish reviews combining supervised and unsupervised approaches," Expert Systems with Applications, vol. 40, pp. 3934-3942, 2013.

[23] [24] [25]

275

M. Thelwall, K. Buckley, G. Paltoglou, D. Cai, and A. Kappas, "Sentiment strength detection in short informal text," Journal of the American Society for Information Science and Technology, vol. 61, pp. 2544-2558, 2010. H. Kang, S. J. Yoo, and D. Han, "Senti-lexicon and improved Naïve Bayes algorithms for sentiment analysis of restaurant reviews," Expert Systems with Applications, vol. 39, pp. 6000-6010, 2012. E. Cambria, "An introduction to concept-level sentiment analysis," in Advances in Soft Computing and Its Applications, ed: Springer, 2013, pp. 478-483. R. Feldman, "Techniques and applications for sentiment analysis," Communications of the ACM, vol. 56, pp. 82-89, 2013. S. A. Kadam and M. S. T. Joglekar, "Sentiment Analysis, an Overview," International Journal of Research in Engineering & Advanced Technology, 1/4, p1, vol. 7, 2013. H. Tang, S. Tan, and X. Cheng, "A survey on sentiment detection of reviews," Expert Systems with Applications, vol. 36, pp. 10760-10773, 2009. G. Vinodhini and R. Chandrasekaran, "Sentiment analysis and opinion mining: a survey," International Journal, vol. 2, 2012. R. Moraes, J. F. Valiati, and W. P. GaviãO Neto, "Document-level sentiment classification: An empirical comparison between SVM and ANN," Expert Systems with Applications, vol. 40, pp. 621-633, 2013. A. Yessenalina, Y. Yue, and C. Cardie, "Multi-level structured models for document-level sentiment classification," in Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, 2010, pp. 1046-1056. V. Jagtap and K. Pawar, "Analysis of different approaches to SentenceLevel Sentiment Classification," International Journal of Scientific Engineering and Technology (ISSN: 2277-1581) Volume, vol. 2, pp. 164-170, 2013. O. Täckström and R. McDonald, "Semi-supervised latent variable models for sentence-level sentiment analysis," in Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies: short papers-Volume 2, 2011, pp. 569574. T. Wilson, J. Wiebe, and P. Hoffmann, "Recognizing contextual polarity in phrase-level sentiment analysis," in Proceedings of the conference on human language technology and empirical methods in natural language processing, 2005, pp. 347-354. V. Singh, R. Piryani, A. Uddin, and P. Waila, "Sentiment analysis of movie reviews: A new feature-based heuristic for aspect-level sentiment classification," in Automation, Computing, Communication, Control and Compressed Sensing (iMac4s), 2013 International MultiConference on, 2013, pp. 712-717. E. Martínez-Cámara, M. T. Martín-Valdivia, and L. A. Ureña-López, "Opinion classification techniques applied to a spanish corpus," in Natural Language Processing and Information Systems, ed: Springer, 2011, pp. 169-176. J. Kamps, M. Marx, R. J. Mokken, and M. De Rijke, "Using wordnet to measure semantic orientations of adjectives," 2004. A. Esuli and F. Sebastiani, "Determining the semantic orientation of terms through gloss classification," in Proceedings of the 14th ACM international conference on Information and knowledge management, 2005, pp. 617-624. A. Esuli and F. Sebastiani, "Sentiwordnet: A publicly available lexical resource for opinion mining," in Proceedings of LREC, 2006, pp. 417422. J. Brooke, "A semantic approach to automated text sentiment analysis," SIMON FRASER UNIVERSITY, 2009. D. Dubois, "Sentiment analysis: Transferring knowledge across domains," 2012. J. Brooke, "Cross-Linguistic Sentiment Analysis : From English to Spanish," pp. 50-54, 2009. N. Samsudin, M. Puteh, A. Hamdan, M. Zakree, and A. Nazri, "Immune Based Feature Selection for Opinion Mining," in Proceedings of the World Congress on Engineering, 2013, pp. 120-127. M. Puteh, N. Isa, S. Puteh, and N. A. Redzuan, "Sentiment Mining of Malay Newspaper (SAMNews) Using Artificial Immune System," in Proceedings of the World Congress on Engineering, 2013.