ISDA 2010

0 downloads 0 Views 143KB Size Report
2016 Carthage Présidence, Tunisie. [email protected]. Abstract— In this paper, we present an approach t 1 o automatically extract and classify opinions in ...
Automatic Extraction and Classification Approach of Opinions in Texts Rihab Bouchlaghem LARODEC, ISG de Tunis 2000 Le Bardo, Tunisia [email protected]

Aymen Elkhlifi Paris-Sorbonne University, 28 rue Serpente, Paris, France [email protected]

Abstract— In this paper, we present an approach t 1 o automatically extract and classify opinions in texts. Thus, we propose a similarity measurement calculating semantically distances between a word and predefined subgroups of seed words. We have evaluated our algorithm on Semantic Evaluation Company “SemEval”, and we obtained the best value of Precision and F1 62 per cent and 61 per cent. As an improvement of 20 per cent compared to others participators. Keywords- Opinion Mining, Natural Language Processing, Semantic Similarity.

I.

INTRODUCTION

With the growing popularity of Web 2.0, the opinion mining becomes more and more interesting. New kinds of opinion-rich sources appeared like blogs, social networks, wikis and online reviews sites. Due to the largest size of this available data, we need to automate their extraction and organization. This requirement has motivated researches to detect opinions in text passages and assign them to subjective classes: positive or negative opinions. Different techniques have been applied to this purpose such as machine learning classifiers generally based on lexical features [1] or syntactic features [2] associated with opinion. A wide range of statistical methods are also investigated in order to detect and classify subjective texts [3, 4]. In this paper, we examine an alternative extraction and classification strategy of opinions based on semantic similarities between terms. Accordingly, we propose to solve the problem of opinion extract in stages; starting with words and moving on to sentences. In fact, we take as unit opinion carrier a single word, and first classify each adjective, verb, and noun by its opinion. Then, we combine the elementary opinion information in order to find the overall opinion class. The rest of the document is organized as follows: section (2) introduces the related works on opinion classification. In section (3), our approach to the extraction and the classification of opinions is described in detail. The experimentation is described in section (4). In section (5) we evaluate the Sec-Op system in order to demonstrate its 1

Rim Faiz LARODEC, IHEC de Carthage, 2016 Carthage Présidence, Tunisie [email protected]

ability. In section (6), we conclude with a few notes on future works. II.

RELATED WORKS ON OPINION CLASSIFICATION

Research in opinion mining includes distinguishing subjective from objective language [2, 5, 6, 1, 7] as well as distinguishing positive from negative language [8, 9, 2, 10]. Various approaches have been adopted to address the first question. For example, Yu and Hatzivassiloglou [8] use unsupervised statistical techniques for detecting opinions at the sentence level. While Bethard, Thornton, Hatzivassiloglou and Jurafsky [1] use a statistical approach divided on two methods for proposition opinion classification. The first method relies on differences in the relative frequency of a word in subjective documents, versus documents that contain mostly facts using the TREC 8, 9, and 11 text collections. The second method used cooccurrence information to extend a seed list of 1336 opinion adjectives. Wilson, Wiebe and Hwa [2] propose an automatic opinion classification approach to classify nested clauses in every sentence in the corpus. They use a wide range of features, including new syntactic features. Although these methods achieve high precision, they are based on a large corpus, and need a large amount of manually tagged training data. Unlike these works, we use only a set of seed words [6]. For word sentiment classification, Hatzivassiloglou and McKeown [9] use a supervised learning algorithm to infer the semantic orientation of adjectives from conjunction constraints. While Turney [4] applies a specific unsupervised learning algorithm based on the mutual information between sentences and the words "excellent" and "poor"), where the mutual information is calculated using statistical techniques. In earlier work [4] only singletons ("excellent" and "poor") were used as seed words. In our work, we use many seed words having more than one subjective category: harm, approval, joy, criticism, pleasure, etc., in order to test whether multiple seed words have a positive effect in extraction performance. In contrast to statistical approaches, symbolic approaches such as proposed by Maurel and Dini [10] and Vernier, Monceaux and Daille [7] provide a better text analysis to represent the grammatical and semantic structure of analyzed text.

We thank the France embassy in Tunis (IFC) for their financing within the framework of doctorate thesis.

1

However, symbolic rules are developed manually, based on annotated data so well. Moreover, such approaches are language-dependents. Our work differs from these approaches in that our method does not need a training corpus to perform the task. Our seed words are also domain independent. We propose to use semantic similarities and synonym sets to extract opinion oriented terms which will be used in general opinion classification. III.

THE PROPOSED APPROACH OF OPINIONS EXTRACTION AND CLASSIFICATION

Our approach is performed in three steps as show Figure1: 1. Preprocessing: it consists on the one hand, in the segmentation of text into segments and, on the other hand, in the POS (Part-Of-Speech) tagging. 2. Opinions-oriented words extraction: is to extract the opinion-oriented words, by calculating their distances compared to subsets of predefined seed words. We propose in this stage a similarity measure between terms. 3. Opinion classification: consists in classifying the polarity of general opinion based on elementary computing of the second stage.

account all typographical markers. Moreover, other linguistic bases are engaged like the syntactic structure of a sentence and the significance of each typographical marker in a well defined context. The existing tools segment the well structured texts into paragraphs. We developed our own segmentor while basing ourselves on punctuation marks. Due to the great number of the linguistic rules to program, we have to integrate in our knowledge base all the rules developed in the system Segatex [11]. The POS tagging: consists in the automatic words annotation with part-of-speech tags, based on both its definition, as well as its context. We used the POS tagger to produce the part-of-speech tag for each word in order to select adjectives, nouns, adverbs and verbs. The following shows a sentence with POS tags. The DT tags are determiners, the JJ tags indicate adjectives, the

RB tags are adverbs, the VVG tags are verbs, and the NN tags are nouns: Sentence: «A thoughtful , provocative , insistently humanizing film» Output tagging: Word A thoughtful provocative insistently humanizing film

POS DT JJ JJ RB VVG NN

Lemma A thoughtful provocative insistently humanize film

Selected Words: Word thoughtful provocative insistently humanize

Figure 1. Proposed approach of opinions extraction and classification

A. Preprocessing In our study, preprocessing comprise the segmentation of text into sentences then the POS tagging of its words. The segmentation: is the determination of the sentences borders. It is a hardly-realizable task. Given that a point followed by a capital letter is not enough to detect the end or the beginning of a segment it is necessary to take into

Syntactic category Adjective Adjective Adverb Verb

B. Opinion-Oriented Words Extraction Our goal in this level is to extract all terms holding opinions and to determine its semantic orientations, which will be used to predict the polarity of global opinion. Our extraction method is based on a polarity score calculated for every term. So, the word is called subjective if its polarity score exceeds a threshold previously fixed. To decide which words are opinion-oriented, we propose an algorithm which assigns to each word a score allowing to determinate its polarity. This algorithm is based on similarity measure developed to this aim. The measure calculates similarities between word and subgroups of semantically oriented words. 1) Measure P-SIM (Polarity-SIMilarity) We put forward a similarity measure computing the semantic similarity of a term compared with others in ontology in order to well classify it.

2

The proposed measure called P-SIM provides similarity quantification between the term and a highly subjective set of words belonging to the same subgroup; the term is represented by itself as well as its synonyms. Then, we define P-SIM between the term (t) and the subgroup (SubSeed), as follows: m n

P - SIM (t, SubSeed) =



∑ ( sim( Syn(t ) , SubSeed i

j =1

i =1

j

)

m n

With: • sim : The similarity measure between one term and an occurrence of SubSeed. • syn(t) : The synonym set of a term t, • m : The size of SubSeed, • n : The size of syn(t) . P-SIM measure provides precise quantification of the term subjectivity, since it identifies relations between a given term and subjective subcategory. For example, applying this measure to the verb "declare" and one of the sub seeds: Agreement having the flowing items: {consent; assent; admit}:

P-SIM ("declare", Agreement) = sim (" declare" , consent) + sim (" declare" , assent) + sim (" declare" , admit)

n



SG = U SubSeed i , a set of predefined seed words



Threshold_Iter : The maximum number of synonyms

1

generation. This threshold serves to prevent the determination of semantic orientations term if there is no variation in his score. • ScoreMin : The minimum score to affecte a term in a class. This threshold is fixed experimentally. • k : The number of generation synonym for a term t. Algorithm: Score_assignment

t, syn(t), SG, k, tab

Input:

Begin Score_assignment

Let's score = 0.9 Let's g = 0 , k = 1

∈ SG) Then score =

If (t

3

score k

(k ≥ threshold _ Iter) Then score = max (tab) //a matrix that store all scores calculated ElseIf

Else

=

For each SubSeed ∈ seed_group

Agreement

0 .63 + 0 .5 + 0 .95

//score initially assigned to terms of subgroups

tabk [g] = P-SIM (t, SubSeed)

= 0, 7

g = g +1 End for

Figure 2. Example of P-SIM computing

2) Score assignment algorithm We concede that a term have the same polarity as their direct synonyms. Otherwise, similar words tend to have the same subjectivity class. We adopt theses hypothesis and we propose an algorithm (see figure 3) by resorting to the similarities between terms and to words synonym sets in order to predict the semantic term orientation. Our strategy is to use a set of predefined seed words. To be able to prepare our seed lists, we undertook an annotation effort of 8000 weak and strong opinion words. Then, we tried to divide this set into subgroups according to subjectivity categories such as: criticism, happiness, harm, approval, joy, etc. The complete procedure for assigning a polarity score to a given word is shown in Figure 4.

( max (tabk ) >= ScoreMin) then

If

score = max (tabk ) Else

syn(t) = synonym(syn(t)) sub_group = synonym (SG) k = k +1 score = Score_assignment (t, SG, k, tab)

End If End If End Score_assignment Ouput:

score

Let’s: •

synonym(t) : a function that generate the synonyms of

Figure 3. Extracting the opinion oriented terms

a term t.

3

Giving an example of score assignment. We applicate our algorithm on the verb as show fowling: t = “declare”; POS = “verb”; threshold_Iter =4; ScoreMin= “0.5” iteration n°1 (k=1): membership (t, seed_group) = false  k < threshold_Iter

 P-SIM (t, SubSeed )

∀SubSeed ∈ seed _ groupe

Sub_groups list: 1. «attraction»; 2. «love»; 3. «desire»; 4. «approval»; 5. «agreement»; 6. «pleasure»; 7. «joy»; 8. «appreciation»; 9. «sympathy»; 10. «goodness»; 11. «criticism»; 12. «disapproval»; 13. «opposition»; 14. «dislike»; 15. «annoyance»; 16. «displeasure»; 17. «pain»; 18. «anxiety»; 19. «sadness»; 20. «misbehavior»; 21. «harm»; 22. «anger»; 23. «blame».

assignment algorithm defined in the preceding section to predict the polarity of global opinion. We calculate a general score based on elementary scores, which can also include adjective modifiers such as negation and quantifiers, besides the verbs and nouns. The polarity is classified according to the sign of the overall score. In fact, the sentences for which the general score is positive are classified as positive opinions, and the sentences with negative scores are classified as negative opinions. >?_

IV.

Similarity (term, sub_group) 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. 15. 16. 17. 18. 19. 20. 21. 22. 23.

Sim (declare, «attraction») Sim (declare, «love») Sim (declare, «desire») Sim (declare, «approval») Sim (declare, «agreement») Sim (declare, «pleasure») Sim (declare, «joy») Sim (declare, «appreciation») Sim (declare ,« sympathy») Sim (declare ,« goodness») Sim (declare, «criticism») Sim (declare , «disapproval») Sim (declare, «y- opposition») Sim (declare, «dislike») Sim (declare, «annoyance») Sim (declare, «displeasure») Sim (declare, «pain») Sim (declare, «anxiety») Sim (declare, «sadness») Sim (declare, «misbehavior») Sim (declare, «harm») Sim (declare, «anger») Sim (declare, «blame»)

=0 = 0.19 = 0.14 = 0.49 = 0.7 =0 = 0.04 = 0.49 = 0.20 = 0.055 = 0.43 = 0.48 = 0.43 =0 = 0.04 =0 =0 = 0.41 = 0.07 = 0.22 = 0.03 =0 = 0.15

max (tab) = 0.7 > ScoreMin  score (declare) = 0.7

EXPERIMENTATION

A system, called Sec-Op (System of extraction and classification of opinions), based on the proposed method has been implemented in Java under Eclipse platform. SecOp includes the four following modules: • Module 1: Text segmentation, • Module 2: POS tagging, • Module 3: Opinion Oriented terms Extraction, • Module 4: Opinion classification. We use our developed module to segment text as described in 2.1. Then, we used the API of Tree tagger [12] to obtain part-of-speech information. We have resort to WordNet [13] for generating synonyms, and to the measures HSO [14] and LIN [15] for computing terms similarities. We have used the SemEval 2007 (the 4th international workshop on Semantic Evaluation) corpus related to Affective text task which is intended as an exploration of the connection between lexical semantics and subjectivity. The corpus consists of 1000 news headlines, extracted from news web sites (such as Google news, CNN) and/or newspapers. V.

RESULTS

To evaluate our approach, we employ the precision and recall measurements [16]. We assign each sentence to one of the three following categories: a: sentence correctly assigned to a given class value, b: sentence not detected (silent), c: sentence misclassified.

Figure 4. Score assignment to the verb «declare»

Firstly, the algorithm tests if the given word belongs to one of subjective subgroups. If so, the polarity score will be the highest. Else, the algorithm resorts to P-SIM for calculating word's similarities to all subgroups. The polarity score receives the maximum value given by P-SIM if this latter is greater than the threshold. If else, the synonyms of the given word and those of seed words are generated, and a recursive call is performed with these new inputs. If maximum iterations number is reached, the word receives the maximum value given by P-SIM across all the iterations. C. Opinion classification As our measure of semantic orientation across an entire sentence, we used the average of scores given by our score

The Precision, the Recall and F1 prove that to be calculated as: a and 2× P× R a , R= P = F1 = a+b a +c (P + R) Table 1 shows the evaluation results of our system: TABLE I.

EVALUATION RESULTS USING SEMEVAL 2007 CORPUS Precision

Recall

F1

Global performances

62.57

62.06

61.27

Positive class

57.38

73

64.25

Negative class

67.76

51.14

58.28

4

We proceeded to compare our results with those of systems participating in the task Affectionate text of SemEvel 2007 [17]. We obtained the best value of Precision and F1 (Precision: 62.57 vs. 61.42; F1: 61.27 vs. 42.43). Table 2 shows the interest of our approach. TABLE II.

COMPARISON OF OUR RESULTS (SEC-OP) AND THE RESULTS OF SEMEVAL2007 PARTICAPTORS Precision

Recall

F1

CLaC

61.42

9.20

16.00

UPAR7

57.54

8.78

15.24

SWAT

45.71

3.42

6.36

CLaC-NB

31.18

66.38

42.43

SICS

28.41

60.17

38.60

SEC-OP

62.57

62.06

61.27

[4]

[5]

[6]

[7]

[8]

[9] [10] [11]

VI.

CONCLUSION AND FUTURE WORKS

In this article, we have proposed an approach for the extraction and the classification of opinions based on semantic similarities between words. The proposed approach is spread over three stages to classify the opinion in a text passage, starting, in a first stage, by the preprocessing that consists in segmenting text and tagging its words. In a second step, an algorithm based on P-SIM allows to extract opinion oriented terms. Finally, the polarity of general opinion is predicted with reference to the extracted terms. We validated our approach on a standard corpus from the evaluation company SemEval 2007. The results obtained are promising compared to those given by the participating systems to SemEval 2007. This approach comes within the framework of the Opinion Mining. Actually, it is being applied in a number of interesting applications like: automatic analyses of product reputations, summarization of customers' reviews about product, analyzing and comparing opinions on the web, following customer preferences. In our future works, we propose to adopt Sec-Op for texts in Arabic language. Indeed, our system supports the Arab characters, and the stages of our approach are independent of the language. But, it remains to provide linguistic resources (Synonyms data base, segmentator, etc.) for the Arabic language. Experimentation was made on a small corpus of 5 articles segmented manually and with a small Synonyms data base to measure.

[12]

[13] [14]

[15]

[16]

[17]

Turney P. D. (2002). "Thumbs up or thumbs down? Semantic orientation applied to unsupervised classification of reviews". Proc. ACL’02. Riloff E. and Wiebe J. (2003). "Learning extraction patterns for subjective expressions". Proc. The Conference on Empirical Methods in Natural Language Processing (EMNLP-2003), pp. 105–112. Riloff E., Wiebe J. and Wilson T. (2003). "Learning subjective nouns using extraction pattern bootstrapping". Proc. The Seventh Conference on Natural Language Learning (CoNLL-03). Vernier M., Monceaux L. and Daille B.1. (2009). "Catégorisation des évaluations dans un corpus de blogs multi-domaine". Revue des nouvelles technologies de l'information 2009, pp. 25. Yu H. and Hatzivassiloglou V. (2003). "Towards answering opinion questions: Separating facts from opinions and identifying the polarity of opinion sentences". Proc. EMNLP’03, pp. 129–136. Hatzivassiloglou V. and McKeown K. R. (1997). "Predicting the semantic orientation of adjectives". Proc. ACL’97, pp.174–181. Maurel S. and Dini L. (2009). "Exploration de corpus pour l’analyse de sentiments". Proc. DEfi Fouille de Textes (DEFT-09), p. 11-23. Mourad Gh. (2002). Analyse informatique des signes typographiques pour la segmentation de textes et l’extraction automatique de citations. Réalisation des Applications informatiques: SegATex et CitaRE. Thèse de doctorat Université Paris-Sorbonne soutenance le 02 novembre 2001. Schmid H. (1994). "Probabilistic Part-Of-Speech Tagging using Decision Trees". Proc, The International Conference on New Methods in Language Processing, pp. 44-49. Miller G. (1995). "WordNet: A lexical database for English". Proc. ACM 38, pp. 39-41. Hirst G. and St-Onge D. (1998). "Lexical chains as representation of context for the detection and correction malapropisms". Proc. C. Fellbaum (Ed.), WordNet: An Electronic Lexical Database, The MIT Press, pp. 305--332. Lin D. (1998). "An information-theoretic definition of similarity". Proc. The 15th International Conference on Machine Learning, Madison. Fellbaum C., Grabowski J. and Landes S. (1998). "Performance and confidence in a semantic annotation task". Proc. C. FELLBAUM, Ed., WordNet: an electronic lexical database, Language, Speech and Communication, The MIT Press. Cambridge, Massachusetts, chapter 9, pp. 216-237. Strapparava C. and Mihalcea R. (2007). "SemEval-2007 Task 14: Affective Text". Proc. The Fourth International Workshop on Semantic Evaluations (SemEval-2007).

REFERENCES [1]

[2] [3]

Bethard S., Yu H., Thornton A., Hatzivassiloglou V. and Jurafsky D. (2004). "Automatic extraction of opinion propositions and their holders". Proc. The Association for the Advancement of Artificial Intelligence (AAAI-04). Wilson T., Wiebe J. and Hwa R. (2004). "Just how mad are you? Finding strong and weak opinion clauses". Proc. AAAI’04. Kim S.-M. and Hovy E. (2004). "Determining the sentiment of opinions". Proc. COLING-04, pp. 1267–1373.

5