Bifurcation - how to Verify and Improve the Effectiveness of Personalized Webbased Systems Bo LENNSTRAND1, Christian PERSSON2, Erik WALLIN3 1 Gotland University, SE-620 67 Visby, Sweden Email:
[email protected] 2 Høgskolen i Gjøvik, 2802 Gjøvik, Norway, Email:
[email protected] 3 Royal Institute of Technology, SE-100 44 Stockholm, Sweden Email:
[email protected] Abstract. The paper deals with a personalization system that utilizes descriptions of information characteristics to perform content based filtering in order to increase stated business objectives. The paper reports from testing of the personalization system in two commercial implementations, the website of Sweden’s third largest daily newspaper and the website of one of Sweden’s largest online recruitment services for the public sector. The results indicate that personalization has a positive influence on the stated business objective – traffic increase. Further, a personalization configuration that gives a high weight to the customer’s behavior in the ongoing session compared to the behavior in previous sessions was favorable to promote traffic increase.
1. Introduction Customer attention is a scarce resource considering that an average customer is exposed to over 3,000 commercial messages per day [1]. It is not possible to take into account all of the information exchanged [2]. The customer will probably neglect most of this information and focus mainly on issues related to his/her interests. Can a computer-based information system be designed to capture the customer’s interests and adapt the information to this? A solution to this problem may be of considerable economical value by reducing marketing and operating costs, probably in proportion to the attention increase. From the customers’ point of view, an increased precision in information retrieval is most likely beneficial. Only a limited part of the total amount of the information generated during interactions between a sender and a receiver is needed in order to describe the process. As Simon [3] stressed, “Most of the complex structures found in the world are enormously redundant, and we can use this redundancy to simplify their description.” According to Simon a selection of the total mass of information can be sufficient to create an adequate description of the object. It is our hypothesis that a sender of information, by taking this into account, can find efficient ways to adapt his message to the customer’s interests. The computer-supported process of adapting an information flow according to the customer’s behavior and giving feedback in real-time is called personalization in this paper. The paper deals with a personalization system that utilizes descriptions of information characteristics (i.e. metadata) to perform content based filtering [4] in order to increase stated business objectives. When the personalization system is used by, e.g., an e-commerce website, the customers’ interactions are documented, interpreted and stored as preference profiles in a continuously updated database. The system uses the preference profile when deciding which offering or content to expose to the customer. The paper also describes a method we call bifurcation to measure and verify the efficiency of the personalization system. The bifurcation function permits different offers and
content to be presented to randomized groups of customers, whose responses are separately recorded and measured by the system.
2. Objectives The paper reports from two business case studies with the objective to test if the effectiveness of personalization efforts on website customers’ behavior could be verified and improved, addressing the following question: • Do personalized web pages give better responses than non-personalized pages?
3. Methodology Most of the methods for measuring the effects of personalization described in the literature are focused on improving the utility for the customer. As described in the following section, the evaluation function of the personalization system used in this research takes the business operator’s standpoint and pragmatically focuses on one or a few measurable business objectives. The paper reports from the testing of the personalization system in real life. The first two commercial implementations are the testing grounds. The first test was done in November 2001 at the website of Svenska Dagbladet, Sweden’s third largest daily newspaper. The second test was done in Aug. 2003 at the website of Offentliga Jobb, a large online recruitment service for the public sector. Further details about the tests are presented in section 5.
4. Technology description 4.1 System overview The personalization system is situated on the commercial operator’s servers and utilizes metadata to perform content based filtering [4] with a matching method of semi-asymmetric distance measure types [5] to adapt information in a non-intrusive way. The personalization system depends on the identification system of the surrounding web system. The personalization system contains a personalization function, the operating tool, and what we call the bifurcation function that is used when measuring the effect of the personalization efforts1. The design of the personalization system has taken notable influence from the Peppers et al. model for one-to-one marketing [6] including four steps as shown in Figure 1.
Figure 1. The Peppers et al. model applied to the system.
1
For more detailed descriptions of the personalization and bifurcation methods, see Lennstrand B., Persson C. & Wallin E.: The Sport Planet Case: Design and Prototyping of a System for Ecommerce and Research. COTIM99 conference, Rhode Island, USA, Sep 26-29, 1999 Wallin E., Persson C. & Lennstrand B.: "Web Metrics - Design Specifications of a Web-based System for Personalization with Bifurcation". TAGA2000 Conference, Colorado Springs, USA, Apr 2-5, 2000
Identify. The objective of the identifying method is to recognize a customer and recall his previous visiting behavior. For some service categories, e.g., banking, identifying oneself by an account number or name is natural. However, in the case of many other services, a request for personal data can cause the customer to leave the site. For this reason it was decided to use the already existing function of cookies [7] for identification in the two case studies here reported. The identification function automatically asks for and sets cookies to the customer’s computer. As an issue of major importance, the question of integrity requires some comments: cookies offer an efficient method to identify computers, but they say nothing about the identity of the person using the computer. The customer can control cookies in different ways: (s)he can erase them from the computer, disable cookies on the web browser, and thus prevent cookies from being set. Taken together, we consider that cookies do not threaten the customer’s personal integrity. However, in the following situations, cookies can cause misinterpretation: • A customer erases his cookies or buys a new computer: The system does not recognize the customer and gives the computer a new identity, and thus ignores the old profiles. • A customer is using one computer at home and another at work: The system creates multiple identities for the same customer. • A computer is used by several different customers: All customers are treated as the same one. The problems caused by such misinterpretations were judged to be minor compared to the effect of asking customers to identify themselves when entering the site. The worst-case scenario is that a customer doesn’t receive the most appropriate personalization. Differentiate. When the customer is identified this way, the personal profiles stored in the customer database are used for differentiation. The system uses two different profile types: 1. Article profile, characterizing each information object on the website (i.e., a joboffering or a news article). 2. User profile, consisting of the accumulated article profiles that the customer has interacted with during present and previous sessions. Interact. When a customer interacts with an information object on the website, his profile is updated by the article’s metadata description. For example, when a customer reads sports articles the customer’s profile is updated with sport as the argument. In this way, the customer’s selection of articles with their inherent values is interpreted as a reflection of the customer’s preferred personal values. Customize. Present information objects from categories that match the customer’s profile. From the database containing the information objects, the personalization system uses the customer’s profile to calculate the best match of information objects to present. 4.2 Bifurcation function The bifurcation function divides the customers stochastically into two or more groups with different personalization configurations. The system is thereby able to present different content offers to randomized groups of customers whose responses are recorded and measured separately. By giving one of the groups a configuration that deactivates the personalization, a control group is created. The customer database and the identification system ensure that a certain customer will be directed to the same group every time (s)he visits the website. The bifurcation system can also be used in a trial-and-error-like way to try different personalization configurations in order to find the best parameter settings when optimizing the effects of the personalization system. See Figure 2.
Figure 2. Schematic view of the bifurcation function.
5. Business Case studies 5.1 Svenska Dagbladet Examples of personalization of news services on the Internet reach back to early experiments in the early 1990s [8]. The NY Times’ Internet edition has successfully been using methods to adapt information to different segments [9]. A popular method of adapting information is content filtering, which refers to the concept of selecting content for a customer based on semantic descriptions of the news items and corresponding customer profiles [10]. Turpeinen et al. [11] and Jokela et al. [12] have also described systems based on this method. Content filtering was used in the implementation at Svenska Dagbladet (SvD), Sweden’s third largest daily newspaper. The personalization system was implemented on the SvD website in August 2001. The personalization operation is based on two metadata descriptions of the news material, Category (e.g., sports, entertainment, international news) and Geographic location (e.g., Stockholm, Sweden, Europe, etc.). For the category description a TT NITF metadata structure was used [13]. This structure originates from the International Press Telecommunications Council’s News Industry Text Format [14]. A journalist who has written an article also encodes it with metadata. SvD’s primary business objective was to increase the average number of page impressions (defined as the total number of downloads of web pages) a customer demands each session. The selection of functions and page areas to personalize was done in cooperation with SvD. One of the requests was that the first page should contain one personalized area (see figure 3) in order to address the problem that the majority of the customers left the site after only viewing the first page. A second personalized area follows at the end of an individual editorial article. The assumption was that when a customer had finished reading an article, a personalized selection of articles presented below it would increase the chance that the customer would choose to read more. During the week from November 19 to 25, 2001, a test was performed. Altogether 81,305 test subjects were registered and divided into four heuristically chosen bifurcation test groups (three personalized and one non-personalized, each with 25 % of the traffic). The customers in the control group received articles selected alphabetically rather than based on individual personalization. The three personalized groups differed according to the weight with which the profile representing the interactions during the active session merged with the profile representing the customer’s historical behavior, i.e., the volatility of the system. This weight was 33 %, 67 % and 100 % respectively. A setting of 100 % provides an entirely session-dependent system or in other words an extremely volatile system.
Figure 3 shows the small personalized area in the upper-right hand corner of the first web page (marked with an ellipse) that was used to display a section name as header in a large font and the heading of the latest editorial article in that section in a smaller font. The smaller text was a hyperlink to the selected article. The articles used were limited to the three most recently published articles from the different sections.
Figure 3. SvD’s first web page
However, it should be noticed that even in the completely volatile group historical behavior determines what is shown the first time on the first page. During the test all interactions performed by the customer were logged. In the test, the customer profiles originated from the beginning of the implementation in August 2001 and were not erased at the start of the test. The results show that the 33 % session weight personalization group is between -4.6 % and 11.6 % better - on average 1 % better - than the non-personalized group with regards to average number of interactions per session. The 67 % session weight group is between 1.3 % and 29.5 % better and on average 10 % better than the reference group. The 100 % session weight group varies between -4.3 % and 19.3 % better with an average of 8 %. All differences between the reference group and all test groups were found to be statistically significant when carrying out the ANOVA test. Kruskal-Wallis’ test also showed a significant group difference. 5.2 Offentliga Jobb The personalization system implementation at the Offentliga Jobb uses two different customer identification mechanisms: anonymous customers get a unique identifier in the form of a cookie set upon his or her computer while registered customers are identified in the log-in procedure. The system uses both methods simultaneously. When a customer enters the website anonymously, the cookie identification determines the personalization experience. If the customer logs in, the registered customer’s profile is activated and merged with the previously anonymously recorded behavior [15]. Offentliga Jobb’s personalization system uses two kinds of metadata profiles for describing the articles (job-offerings), a standard format called SSYK for classification of professions and a geographical classification. The personalization system was implemented in May 2002. Offentliga Jobb’s business objectives were to increase the ease-of-use and satisfaction of the customers and to increase the traffic on the website. (In this study, only the increase of traffic has been investigated.)
On the first page the navigation of geographical and job categories were personalized with preset drop-down menus. Figure 4 shows that the personalization system has preset the menus with Stockholm and healthcare (ellipse A). The visitor still has the opportunity to change the selection by choosing another alternative. On the first page the four most appropriate job offerings (according to the visitor’s past behavior) also were shown. Frame B contains four job offerings associated with Stockholm and healthcare were presented. The selection of job offerings was limited to the two most recent weeks.
Figure 4. Offentliga Jobb’s first web page (personalized area marked)
The following personalization functions were selected in cooperation with Offentliga Jobb: A. First web page navigation B. First web page selection of job offerings C. Top two selections in the search results presentation D. Small advertisements in the side bar A test of the personalization system was performed during six weeks at Offentliga Jobb’s website. The test started on June 30, 2003, about a year after the system first was implemented, and ended on August 10, 2003. The test was limited to one bifurcation group with active personalization getting 90 % of the traffic and one control group with no personalization with 10 % of the traffic. The control group received the articles and offerings selected chronologically instead of personalized. The test structure was made at the request of the company, which wanted the personalization system to function for a maximum number of customers. The total number of sessions with interactions where a customer clicked on an object known to the personalization system was 43,002. The personalized group had an increase in interactions of 6,1% compared to the control group with regards to average number of interactions per session. The t-test as well as the Mann-Whitney U-test indicated that this difference was statistically significant. 5.3. Results The bifurcation function provided a purposeful tool for the evaluation of the personalization system implementations. The case studies indicate that personalization has a positive influence on the stated business objective, i.e., traffic increases. Bearing in mind that the test periods were short, the following results should be interpreted carefully and treated as indicative: I. The personalized groups’ average number of interactions increased compared with the non-personalized group in both case studies.
II. A personalization configuration that gives a relatively high weight to the customer’s behavior in the ongoing session compared to the behavior in previous sessions was favorable to promote a traffic increase. The registered increase of traffic is, although statistically significant, relatively small, 10% on average in the SvD case. This may be due to the fact that only a minor part of the web pages were personalized. Most of the content exposed was not personalized. The business objective to increase traffic seems to work more effectively when the customer is driven by broad interests (reading news articles) than when a customer is actively seeking information on a focused issue (job search). Still, an increase in traffic could be seen also in the Offentliga Jobb case. However, when looking at the results from the Offentliga Jobb case, an appropriate job selection presented on the first web page should facilitate the customer’s search, thereby actually leading to a decrease in the traffic. If a customer finds the job offer (s)he looks for on the first page, no more interactions are needed. Therefore, it may not be optimal to use traffic increase as a measure of the fulfillment of the business objectives. From this, we conclude that the bifurcation function seems to be helpful when evaluating the performance of a personalized website, but other measures than the number of interactions should also be developed.
6. Industrial significance and benefits The two case studies indicate that adapting information according to individual customer preferences can increase traffic. Additional installations of the personalization system at International Data Group Sweden (publishing house), Retail and Brands (fashion retailer), and Arena Personal (recruitment) confirm the commercial value and the results of the SvD and Offentliga Jobb cases. The increase in traffic indicates that an increase in usability and customer satisfaction has occurred. This makes personalization an interesting concept for all kinds of web information and not limited only to commercial connections. Personalization can be a useful concept also for public information delivered by the Internet. Generally speaking personalization can be a useful tool in all situations where information is to meet with personal preferences on the web or in other digital environments, i.e., the mobile telecom nets or the digital TV nets. This is especially the case in those situations where information overload is a considerable risk. The bifurcation function provides a tool to measure the business impact and creates an opportunity to optimize the performance. The personalization system described is easily integrated with an existing system. A typical integration takes less than 100 hours. The personalization system is independent of the mechanism for identifying the customers, database types or content management systems as long as they support Microsoft, Java or PHP technologies. To secure the integration process and the continuous operations of the personalization system the supporting documentation was done using the process methods of Rational Unified Process. The personalization system relies on the categorization of the objects to be adapted. It is of uttermost importance that the categorization is done consistently and that the categorization structure has relevance to the objectives of the personalization effort.
7. Future research There are numerous aspects ranging from social science to business science via computer science to explore in the information adaptation domain. Further investigations are needed to determine which functions that can be personalized, how they can be designed, and where they can be placed. Additional research is needed on how to optimize the configurations and matching algorithms of the personalization system and how to find out what the
optimal categorization structure is, as well as longitudinal tests for confirming the effects of personalization are eligible. Finding and using other quantitatively measurable business objectives such as return frequency (i.e., loyalty) is an important research field. Using qualitative methods such as interviews, questionnaires, and observations, etc., together with the bifurcation function to study the impact of personalization systems on both the business operator’s objectives and the customers’ utility will be essential.
References [1] M. Landler,W. Konrad, Z. Schiller, L. Therrien, WHAT HAPPENED TO ADVERTISING. Business Week September 23, New York, 1991. [2] P. West, D. Ariely, S. Bellman, E.Bradlow, J. Huber, E. Johnson, B.Kahn, J. Little, D. Schkade, Agents to the Rescue? Marketing Letters, Vol. 10:3, 1999, pp. 285-300. [3] H. Simon, The Sciences of the Artificial. The MIT Press, Cambridge, 1998, p.215. [4] P. Foltz, S. Dumais, Personalized Information Delivery: An Analysis of Information Filtering Methods. Communications of the ACM, Vol 35:12, 1992, pp.51-60. [5] E. Savia, T. Kurki, S. Jokela, Metadata Based Matching of Documents and User Profiles. Proceedings of the 8th Finnish Artificial Intelligence Conference, Human and Artificial Information Processing, Finnish Artificial Intelligence Society, Jyväskylä, Finland, 1998, pp.61-70. [6] D. Peppers, M. Rogers, B. Dorf, The one to one fieldbook: The Complete Toolkit for Implementing a 1to1 Marketing Program. Doubleday, New York, 1999. [7] D. Kristol, L. Montulli, HTTP State Management Mechanism. Bell Laboratories. Lucent Technologies, Netscape Communications, RFC2109, 1997. [8] P. Chesnais, M. Mucklo, J. Sheena, The Fishwrap Personalized News System.In the proceedings of the IEEE 2nd International Workshop on Community Networking Integrating Multimedia Services to the Home, Princetown, 1995. [9] C. Saunders, NYTD Points to “Surround Session” Benefits. www.internetnews.com /IAR/article.php/12_970991 accessed Dec.8, 2003. [10] O. Lassila, Introduction to RDF Metadata. W3C, www.v3c.org/TR/NOTE-rdf-simple-intro-971113.html accessed Dec. 8, 2003. [11] M. Turpeinen, J. Saarela, M. Korkea-aho, T. Puskala, R. Sulonen, Architecture for Agent-Mediated Personalised News Services. PAAM’s 96: The First International Conference on Practical Application of Intelligent Agents and Multi-Agent Technology, London, 1996. [12] S. Jokela, M. Turpeinen, T. Kurki, E. Savia, R. Sulonen, The Role of Structured Content in a Personalized News Service. Proceedings of the 34th HICCS, Computer Society Press, 2001. [13] H. Stadler, J. Lindgren, TT NITF version 3.3. Tidningarnas Telegrambyrå, www.tt.se/ tekspec/ttnitfv3_3.doc accessed Dec. 8, 2003. [14] A.Karben, Recent Modifications. Most Recent Changes to the NITF DTD. www.nitf.org/site/recentmodifications.html accessed Dec. 8, 2003. [15] E.Wallin, Impact of Content-Based Personalization in a Service Environment. A Case Study. 5th Annual Conference on World Wide Web Applications, Durban, 2003.
