Revisiting Rocchio's Relevance Feedback Algorithm ... - Springer Link

Revisiting Rocchio’s Relevance Feedback Algorithm for Probabilistic Models Zheng Ye1,2 , Ben He2 , Xiangji Huang2 , and Hongfei Lin1 1

Department of Computer Science and Engineering, Dalian University of Technology Dalian, Liaoning, 116023, China 2 Information Retrieval and Knowledge Managment Lab, York University, Toronto, Canada [email protected],{benhe,jhuang}@yorku.ca,[email protected]

Abstract. Rocchio’s relevance feedback method enhances the retrieval performance of the classical vector space model. However, its application to the probabilistic models is not adequately explored. In this paper, we revisit Rocchio’s algorithm by proposing to integrate this classical feedback method into the divergence from randomness (DFR) probabilistic framework for pseudo relevance feedback (PRF). Such an integration is denoted by RocDFR in this paper. In addition, we further improve RocDFR’s robustness by proposing a quality-biased feedback method, called QRocDFR. Extensive experiments on standard TREC test collections show that our proposed RocDFR and QRocDFR methods significantly outperform the relevance model (RM3), which is a representative feedback model in the language modeling framework. Moreover, the QRocDFR method considerably improves the robustness of RocDFR’s retrieval performance with respect to the size of feedback document set. Keywords: Information Retrieval, Feedback, DFR, QRocDFR.

1

Probability

Model,

Relevance

Introduction

In information retrieval (IR), relevance feedback (RF) can improve query representation by taking feedback information into account. A classical relevance feedback algorithm was proposed by Rocchio in 1971 [14] for the SMART retrieval system [14]. It takes a set of documents as the feedback set. Unique terms in this set are ranked in descending order of TF-IDF weights. A number of top-ranked candidate terms are then added to the original query, and finally, documents are returned for the expanded query. Better performance can always be expected in many IR task. Many other relevance feedback techniques and algorithms have been developed, mostly derived from Rocchio’s algorithm [1,3,12,13]. Feedback documents can be obtained by many possible means. In general, there are explicit evidence, such as the labeled relevant documents from real users, or implicit evidence, such as the click-through data. Obtaining the feedback information involves extra efforts, e.g. real user relevance judgment, and is usually expensive. For every P.-J. Cheng et al. (Eds.): AIRS 2010, LNCS 6458, pp. 151–161, 2010. c Springer-Verlag Berlin Heidelberg 2010

152

Z. Ye et al.

given query, the corresponding feedback information is not necessarily available. An alternate solution is query expansion (QE), also called pseudo-relevance feedback (PRF), which uses the top-ranked documents in the initial retrieval for the feedback [5]. Its basic idea is to extract expansion terms from the top-ranked documents to formulate a new query for a second round retrieval. Despite the marked improvement brought by Rocchio’s relevance feedback algorithm (RocRF), the overall retrieval performance of RocRF over the vector space model (VSM) [15] is still not satisfying enough as shown in [16]. In particular, with the recent development of feedback methods [7,16], within the language modeling framework [11], such a combination of RocRF with VSM can no longer catch up with the state-of-the-art methods. In [8], Lv et al. systematically compare five state-of-the-art approaches for estimating query language models in ad-hoc retrieval, in which an instantiation of the relevance model [7], called RM3, not only yields effective retrieval performance in both precision and recall metric, but also performs robustly with different feedback document set sizes. The aim of this paper is to investigate ways to improve PRF for probabilistic models. Our main contributions are as follows. First, we propose a new PRF paradigm, called RocDFR, by revisiting and adopting the classical Rocchio’s relevance feedback framework within a divergence from randomness (DFR) probabilistic framework [1]. Second, we further enhance the robustness of RocDFR by introducing a quality-biased feedback method, called QRocDFR. Finally, extensive experiments also show that the retrieval performance of our proposed RocDFR and QRocDFR significantly outperform the state-of-the-art relevance model within the language modeling framework. The remainder of this paper is organized as follows. Section 2 surveys previous work on PRF. Section 3 introduces our proposed PRF method adapted for the DFR framework. Section 4 describe our experimental methodology, and Section 5 reports the experimental results and provides the related discussion. Finally, Section 6 concludes on the work and suggested future research directions.

2 2.1

Related Work Rocchio’s Relevance Feedback Method

Rocchio’s algorithm [14] is a classic framework for implementing (pseudo) relevance feedback via improving the query representation. It models a way of incorporating (pseudo) relevance feedback information into the vector space model (VSM) in IR. In case of pseudo relevance feedback, Rocchio’s method (RocRF) has the following steps: 1. All documents are ranked for the given query using a particular Information Retrieval model, for example the vector space model [15]. This step is called first-pass retrieval. The R highest ranked documents are identified as the pseudo relevance set DR .

Revisiting Rocchio’s Relevance Feedback Algorithm for Probabilistic Models

153

2. An expansion weight w(t, DR ) is assigned to each term appearing in the set of the R highest ranked documents. In general, w(t, DR ) is the mean of the weights provided by the IR model, for example the TF-IDF weights [15], computed over the set of the R highest ranked documents. 3. The vector of query terms weight is finally modified by taking a linear combination of the initial query term weights with the expansion weight w(t, DR ) as follows: r (1) Q1 = α ∗ Q0 + β ∗ R r∈DR

where Q0 and Q1 represent the original and first iteration query vectors, DR is the set of pseudo relevance documents, r is the expansion term weight vector, and α and β are tuning constants controlling how much we rely on the original query and the feedback information. In practice, we can always fix α at 1, and only study β in order to get better performance. 2.2

The DFR Probabilistic Framework

Divergence from Randomness (DFR) is a componential framework that measures the relevance of documents following the probabilistic paradigm [1]. In the DFR framework, the weight of a document d for a given query term t is given by: w(d, t) = qtw(t) · IG · (−log2 P rob(tf ))

(2)

where IG is the information gain, which is given by a conditional probability of success of encountering a further token of a given word in a given document on the basis of the statistics on the retrieved set. Prob(tf ) is the probability of observing the document d given tf occurrences of the query term t. −log2 P rob(tf ) measures the amount of information that term t carries in d. qtw is the query term weight component. Similarly to the query model in language modeling [11], qtw measures the importance of individual query terms. In the DFR framework, the query term weight is given by: qtw(t) =

qtf (t) qtfmax

(3)

where qtf(t) is the query term frequency of t, namely the number of occurrences of t in the query. qtfmax is the maximum query term frequency in the query. For example, if all query terms appear only once in the query, which is usually the case for short queries, qtfmax is 1 and qtw(t) is also 1 for each query term. When PRF is applied, qtw(t) is updated according to each query term’s importance in the pseudo relevance documents, so that informative query terms can be differentiated from the non-informative ones. The other two components, namely information gain (IG) and information amount (−log2 P rob(tf )), can be approximated by different statistics such as Poisson distribution and Bose-Einstein statistics, etc. [1]. In this paper, we apply the DPH instantiation of the DFR framework, which is based on the hypergeometric approximation of the term distribution [2,6]. Using the DPH model,

154

Z. Ye et al.

the relevance score of a document d for a query Q can be found in [2]. It is of note that no parameter tuning is required to optimize DPH, and we can rather focus on studying PRF. The DFR framework provides a componential paradigm to which relevance feedback methods can be easily embedded. In the next section, we describe our method of combining the classical Rocchio’s relevance feedback method with the DFR framework.

3

Applying RocRF for DFR Models

In this section, we first demonstrate how to employ the Rocchio’s algorithm (RocRF) within the DFR probabilistic framework. The resulting feedback method is denoted by RocDFR. Next, We propose a quality-biased feedback method to enhance the robustness of RocDFR. 3.1

Adopting RocRF for DFR

We propose to integrate Rocchio’s relevance feedback (RocRF) method into the divergence from randomness (DFR) probabilistic weighting framework. The proposed method updates the query term weight component of the DFR framework by considering an expansion term’s importance in the pseudo relevance set. Our proposed method, called RocDFR, has the follows steps: 1. For a given query, performs the first-pass retrieval and considers the R highest ranked documents as the pseudo feedback set DR . This is the same as step 1 of RocRF described in Section 2.1. 2. Assigns an expansion weight w(t, DR ) to each unique term t in DR , where w(t, DR ) = w(t, dR )/R is the mean of the expansion weights in dR ∈DR each individual feedback document dR . Unlike step 2 of RocRF in Section 2.1, we apply the document ranking scores from the first-pass retrieval to compute the expansion weight w(t, dR ). 3. Finally, the T most weighted expansion terms are added to the query. In particular, the query term weight vector is updated using Equation 1. Above we have proposed a standard deployment of the classical RocRF in the DFR probabilistic framework. One of the obstacles that prevents RocRF’s from its application to probabilistic models is the robustness problem. The retrieval performance of RocDFR could be highly sensitive to its parameters, in particular the size of the pseudo relevance set. In the next section, we propose a qualitybiased feedback method that aims to improve the robustness of RocDFR. 3.2

Quality-Biased PRF

The RocDFR method is simple and computationally efficient, and is usually effective when its parameters are properly optimized, as shown later in our experiments. However, as RocDFR treats each feedback document equally, regardless


155

of their quality, its retrieval performance is likely to be sensitive to the size of the pseudo feedback document set. In particular, as shown in our experiments in Section 5.2, when the feedback document set size is very large, the retrieval performance is seriously degraded by the low-quality feedback documents on some test collections. To this end, we propose a quality-biased pseudo relevance feedback (PRF) method, denoted by QRocDFR, to promote expansion terms in the high-quality documents, and penalize those in the low-quality documents. In our proposed QRocDFR method, a quality-biased factor is introduced to the query term weight updating formula (i.e. Equation 1) as follows: r ∗ q(dr ) (4) Q1 = α ∗ Q0 + β ∗ R r∈R

where q(dr ) is the quality score of feedback document dr in the R highest ranked documents in the first-pass retrieval. The document quality score is given by the sum of the expansion weight of the original query as q(dr ) = t∈Q w(t, dr ). In practice, our proposed QRocDFR method is very flexible since the weight of each feedback document can be determined in different ways. If a uniform quality score is assigned to all feedback documents, RocDFR can be viewed as a special case of QRocDFR. In this paper, we choose to infer the document quality by the expansion weight of the original query terms in the document. The rationale behind our choice of document quality score is that a high-quality feedback document is highly related to the query topic, and hence likely to yield the original query terms. Therefore, our document quality score is given by the sum of the expansion weight of the original query as q(dr ) = t∈Q w(t, dr ).

4

Experimental Setup

4.1

Test Collections and Evaluation

In this section, we describe three representative test collections used in our experiments: disk4&5 (no CR), WT10G, and GOV2. These three collections are different in both size and genre. The TREC tasks and topic numbers associated with each collection are presented in Table 1. Table 1. The TREC tasks and topic numbers associated with each collection Collection Task disk4&5 2004, Robust WT10G TREC9, 10, Web ad-hoc GOV2 TREC04-06, Web ad-hoc

Queries Docs 301-450 528,155 451-550 1,692,096 701-850 25,178,548

In all our experiments1 , we only use the title field of the TREC queries for retrieval. In the process of indexing and querying, each term is stemmed using Porter’s English stemmer, and standard stopwords are removed. The MAP 1

The experiments are conducted with an in-house integration of Terrier 3.0[10] and Lucene (http://lucene.apache.org/)

156

Z. Ye et al.

(Mean Average Precision) performance measure for top 1000 documents is used as evaluation metric. 4.2

Baseline Models and Parameter Training

In order to evaluate the performance of our proposed methods, the baseline used is DFR-RF, the DFR relevance feedback method [1] implemented in the Terrier retrieval platform [9]. The DFR-RF method considers the R highest ranked documents as the pseudo feedback document set DR . Differently from RocDFR, DFR-RF views the R feedback documents as a whole bag of words, and measures the divergence of an expansion term’s distribution in DR from its distribution in the entire document collection. As shown by previous experiments, the DFR-RF method’s effectiveness is highly sensitive to the feedback document set size R [6]. Thus, we expect our proposed methods to outperform DFR-RF in terms of both effectiveness and robustness. In addition, we also compare our proposed methods with relevance model [7], which is a representative and state-of-the-art approach for estimating query language models within language modeling framework[8]. Relevance models do not explicitly model the relevant or pseudo-relevant document. Instead,they model a more generalized notion of relevance R. The formula of RM1 is: p(w|θD )p(θD )P (Q|θD ) (5) p(w|R) ∝ θD

The relevance model p(w|R) is often used to estimate the feedback model θF , and then interpolated with the original query model θQ in order to improve its estimation. The interpolated version of relevance model is called RM3. In order to find the optimal parameter setting for fair comparisons, we use the training method in [4] for both the baseline and our proposed approach. In particular, first, for the smoothing parameter μ in LM with Dirichlet prior, we sweep over values from 300 to 1500 with an interval of 50. Second, for the linear combination parameter β of (Q)RocDFR, and the interpolation parameter α of RM3, we sweep over values in the range of (0.0, 0.1, . . . , 1.0). For the number of expansion terms, we sweep over values in (10, 20, 30, 40, 50). To evaluate the baseline and our proposed approach, we use 2-fold cross-validation, in which the TREC queries are partitioned by the parity of queries number on each collection. Then, the parameters learned on the training set are applied to the test set for evaluation purpose.

5 5.1

Experimental Results and Analyses Performance of Basic Retrieval Models

Table 2 presents the results of the basic retrieval models. In genereal, LM slightly outperforms DPH on all the test collections. However, there is no signicant difference observed, according to the Wilcoxon matched-pairs signed-ranks test at


157

Table 2. Performance of basic retrieval models Basic Models disk4&5 WT10G GOV2 DPH 0.2178 0.1961 0.2994 LM 0.2254 0.2077 0.3040

the 0.05 level. It is of note that the results from LM are obtained under optimal parameter, while DPH is a parameter-free model. In this sense, DPH is still a good basic retrieval model. In addition, it is fair to make cross-comparison of different PRF methods in language modeling framework and the DFR framework respectively, since the basic retrieval models are comparable. 5.2

Comparison of the PRF Methods

Tables 3, 4 & 5 present the comparisons of the PRF methods over different numbers of feedback documents. A star and a “+” indicate a statistically significant improvement over the baselines of DFR-RF and RM3 respectively, according to the Wilcoxon matched-pairs signed-ranks test at the 0.05 level. Generally, as we can see from Tables 3 4 5, QRocDFR is at lease comparable with DFR-RF and RM3 on all the test collections. In some cases, QRocDFR Table 3. Comparison of different PRF methods On the disk4&5 collection. The values in the parentheses are the improvements over DFR-RF and RM3. |Df | 3 5 10 15 20 30 50

DFR-RF RM3 0.2486 0.2517 0.2511 0.2525 0.2552 0.2534 0.2605 0.2522 0.2506 0.2523 0.2460 0.2506 0.2346 0.2495

RocDFR 0.2476 (-%0.4, -1.6%) 0.2511 (0%, -0.5%) 0.2641∗+ (3.37%, 4.05%) 0.2632+ (1.04%, 4.36%) 0.2615∗+ (4.35%, 3.64%) 0.2531 (2.89%, 1%) 0.2466∗ (5.12%, −1.16%)

QRocDFR 0.2484 (0%, -1.3%) 0.2513 (0.1%, 0.46%) 0.2556 (0.16%, 0.86%) 0.2583 (- 0.84%, 2.42%) 0.2612∗+ (4.23%, 3.52%) 0.2627∗+ (6.79%, 4.83%) 0.2598∗+ (10.7%, 4.13%)

Table 4. Comparison of different PRF methods on the WT10G collection. The values in the parentheses are the improvements over DFR-RF and RM3. |Df | 3 5 10 15 20 30 50

DFR-RF RM3 0.2136 0.2215 0.2085 0.2185 0.2067 0.2169 0.1959 0.2156 0.1941 0.2153 0.1945 0.2167 0.1922 0.2164

RocDFR 0.2336∗+ (9.36%, 5.46%) 0.2248 (1.53%, 2.88%) 0.2053 (-0.67%, -5.35%) 0.2113∗ (7.86%, −1.99%) 0.2091∗ (7.73%, −2.88%) 0.2015∗ (8.68%, −7.01%) 0.1985∗ (3.28%, −8.27%)

QRocDFR 0.2348∗+ (9.83%, 6.00%) 0.2329∗+ (11.7%, 6.59%) 0.2337∗+ (12.67%, 7.38%) 0.2278∗+ (16.28%, 5.66%) 0.2285∗+ (17.72%, 6.13%) 0.2260∗+ (17.46%, 4.29%) 0.2196∗ (14.25%, 1.15%)

158

Z. Ye et al.

Table 5. Comparison of different PRF methods on the GOV2 collection. The values in the parentheses are the improvements over DFR-RF and RM3. |Df | 3 5 10 15 20 30 50

DFR-RF RM3 0.3426 0.3287 0.3454 0.3262 0.3327 0.3333 0.3323 0.3330 0.3264 0.3331 0.3289 0.3328 0.3211 0.3322

RocDFR 0.3354(−2.13%, 2.00%) 0.3441(−0.32%, 5.55%) 0.3440∗+ (3.40%, 3.21%) 0.3463∗+ (4.21%, 3.99%) 0.3462∗+ (6.07%, 3.93%) 0.3405∗ (3.53%, 2.31%) 0.3332∗ (3.77%, 0.30%)

QRocDFR 0.3494+ (2.80%, 6.29%) 0.3496+ (1.21%, 7.17%) 0.3562∗+ (5.08%, 4.89%) 0.3592∗+ (8.09%, 7.87) 0.3576∗+ (9.56%, 7.36%) 0.3571∗+ (8.42%, 7.15%) 0.3432∗ (6.88%, 3.31%)

Fig. 1. Robustness of the PRF methods over the number of feedback documents

significantly outperforms both DFR-RF and RM3, especially when a larger number of documents are used for PRF. In addition, QRocDFR and RM3 are more robust with respect to the parameter R, compared with DFR-RF and RocDFR. This is because both QRocDFR and RM3 take into account the quality of the feedback documents in some ways. When low-quality documents are used for PRF, it is likely that noisy terms will be introduced into the query such that the IR performance can be decreased. A more clear picture of the robustness is depicted in Figure 1. While the performance of RM3 is the most robust on all the three collection, QRocDFR achieves better performance than RM3 does in terms of the MAP measure. It is also of note that when a larger number of documents are used for feedback, QRocDFR still achieves relatively good performance. In addition, on the GOV2 collection, we can see that DFR-RF and RocDFR perform stably over parameter R, although DFR-RF and RocDFR view each document the same. This is probably because most of the top 50 documents are positive for PRF. On WT10G and disk4&5, when R approaches 50, the MAP values of both DFR-RF and RocDFR drop dramatically. In the contrast, QRocDFR and RM3 remain stably to some extent. It indicates that the quality of the feedback documents is importance in the process of PRF.


5.3

159

Influence of the Control Parameter β

Recall that we incorporate the feedback information to derive a better representation of the query as shown in Equation (4). How much we rely on the feedback information is controlled by the parameter β. In our preliminary experiments, we found that the control parameter β in Equation (4) plays an important role in obtaining good performance. In this section, we empirically study the influence of this parameter on all the test collections. In particular, Figure 2 depicts its influence over different numbers of feedback documents. In this set of experiments, we use 50 terms to expand the original query since QRocDFR archives very good performance under this setting generally. The MAP at β = 0 is actually the baseline of DPH without query expansion. It is of note that when β takes very large value, it approaches the performance resulting from using only the feedback information. As we can see from Figure 2, in general, the IR performance can always be boosted when the feedback documents are used to expand the original query. Although the setting of β can affect the retrieval performance significantly, it is always safe to set α to a value around 1 on all our test collections. Similar results can be observed over other numbers of feedback documents, which are not presented as the problem of space limitation.

Fig. 2. Influence of the control parameter β

6

Conclusions and Future Work

In this paper, we have revisited the Rocchio’s relevance feedback method and integrated it into the divergence from randomness (DFR) probabilistic retrieval models. Moreover, we have proposed a quality-biased feedback method, called QRocDFR, which takes into account the quality of the feedback document. Extensive experiments on three representative TREC test collections show that QRocDFR is not only effective, but also robust with respect to the number of feedback documents. To some extent, we have shown that the classical Rocchio’s relevance feedback method can be successfully applied to probabilistic models,

160

Z. Ye et al.

and the resulting retrieval performance is at lease as good as those in the existing literature of pseudo relevance feedback methods. In the future, we plan to study how to automatically determine the control parameter β. Although it is often relatively safe to set β to a value around 1, an automatic way to optimizing β for each individual query sounds more interesting. We also plan to use machine learning techniques to estimate the quality of the feedback documents.

Acknowledgments This research is jointly supported by NSERC of Canada, the Early Researcher/Premier’s Research Excellence Award, the Natural Science Foundation of China (No.60673039 and 60973068) , the National High Tech Research and Development Plan of China (No.2006AA01Z151), National Social Science Foundation of China (No.08BTQ025), the Project Sponsored by the Scientific Research Foundation for the Returned Overseas Chinese Scholars, State Education Ministry and The Research Fund for the Doctoral Program of Higher Education (No.20090041110002).

References 1. Amati, G.: Probabilistic models for information retrieval based on divergence from randomness. Ph.D. thesis, Department of Computing Science, University of Glasgow (2003) 2. Amati, G., Ambrosi, E., Bianchi, M., Gaibisso, C., Gambosi, G.: FUB, IASI-CNR and University of Tor Vergata at TREC 2007 blog track. In: Voorhees, E.M., Buckland, L.P. (eds.) TREC. National Institute of Standards and Technology (NIST), vol. Special Publication 500–274 (2007) 3. Carpineto, C., de Mori, R., Romano, G., Bigi, B.: An information-theoretic approach to automatic query expansion. ACM Trans. Inf. Syst. 19(1), 1–27 (2001) 4. Diaz, F., Metzler, D.: Improving the estimation of relevance models using large external corpora. In: Efthimiadis, E.N., Dumais, S.T., Hawking, D., Järvelin, K. (eds.) SIGIR, pp. 154–161. ACM, New York (2006) 5. He, B.: Query expansion models. In: Encyclopedia of Database Systems, pp. 2257– 2260 (2009) 6. He, B., Macdonald, C., Ounis, I., Peng, J., Santos, R.L.T.: University of glasgow at trec 2008: Experiments in blog, enterprise, and relevance feedback tracks with terrier. In: Voorhees, E.M., Buckland, L.P. (eds.) TREC. National Institute of Standards and Technology (NIST), Special Publication 500–277 (2008) 7. Lavrenko, V., Croft, W.B.: Relevance-based language models. In: Croft, W.B., Harper, D.J., Kraft, D.H., Zobel, J. (eds.) SIGIR, pp. 120–127. ACM, New York (2001) 8. Lv, Y., Zhai, C.: A comparative study of methods for estimating query language models with pseudo feedback. In: CIKM, pp. 1895–1898 (2009) 9. Ounis, I., Amati, G., Plachouras, V., He, B., Macdonald, C., Johnson, D.: Terrier information retrieval platform. In: Losada, D.E., Fern´ andez-Luna, J.M. (eds.) ECIR 2005. LNCS, vol. 3408, pp. 517–519. Springer, Heidelberg (2005)


161

10. Ounis, I., Amati, G., Plachouras, V., He, B., Macdonald, C., Lioma, C.: Terrier: A high performance and scalable information retrieval platform. In: Proceedings of ACM SIGIR 2006 Workshop on Open Source Information Retrieval, OSIR 2006 (2006) 11. Ponte, J.M., Croft, W.B.: A language modeling approach to information retrieval. In: SIGIR, pp. 275–281. ACM, New York (1998) 12. Robertson, S.E.: On term selection for query expansion. Journal of Documentation 46(4), 359–364 (1990) 13. Robertson, S.E., Walker, S., Hancock-Beaulieu, M., Gatford, M., Payne, A.: Okapi at TREC-4. In: TREC, pp. 73–97 (1995) 14. Rocchio, J.: Relevance feedback in information retrieval, pp. 313–323. PrenticeHall, Englewood Cliffs (1971) 15. Salton, G., Buckley, C.: Improving retrieval performance by relevance feedback. Journal of the American Society for Information Science 41, 288–297 (1990) 16. Zhai, C., Lafferty, J.D.: Model-based feedback in the language modeling approach to information retrieval. In: CIKM, pp. 403–410 (2001)