Customised Information Delivery: A SIGIR 99 Workshop

Introduction
Sessions
Adaptive Presentation
User Profile Information Systems
Answer Organisation
Adaptive Hypertext
Discussion of Topics Arising
User Modelling
Adaptive Generation
Evaluation
Conclusions
Contacts


Introduction

A major strength of electronic information is the fact that as opposed to its paper counterpart, it can be dynamically modified and tailored to a user's need. The problem has been tackled from different angles by different communities. The IR community has revisited its traditional model of query/answers, and now allows for user interaction, synthesized answers, contextual information when presenting answers, etc. The language technology community has designed user models and natural language interfaces to allow better communication and understanding between the system and the user. The Web community has taken a pragmatic approach, with scripts and virtual documents to produce Web pages.

The aim of this workshop was to bring together these communities, in the hope of fostering collaboration and exchange of ideas. Indeed, the 45 participants to our workshop came from different disciplines: Information Retrieval, Filtering, Natural Language Generation, Human-Computer Interaction, Information Science, and the Web community.


Sessions

The workshop was divided in four sessions: Adaptive Presentation, User Profile Information Systems, Answer Organisation and Adaptive Hypertext. Each session comprised two papers and an "invited talk". There was 30 minutes for discussion at the end of each session.

Adaptive Presentation

In this session we discussed systems that focus on adapting the content to the user. Caroline Eastman introduced the concept of Customizable Information Components, which are basic chunks of information that can be derived and combined to form documents. An hypercard implementation and user experiments were presented. John Tait presented SYSTAR, a Syntactic Simplification System for Aphasic Readers. Their proposal is to help aphasic readers (a reading disorder) by simplifying the original text, for example resolving anaphora. In her position paper, Cécile Paris summarised Adaptive Presentation as a communication problem between the user and the system, and then explained how natural language generation and user modelling can be used to achieve a presentation that "make sense" to the users in their interaction with the system.

User Profile Information Systems

The next session was about user modelling. Two systems that make use of a user profile were presented. Eric Lease Morgan presented MyLibrary@NCState, a digital library that can be customised by users. The system allows users to directly add their own links to the pages, and can present different information according to the users' discipline. Ricardo Mazza presented SwissCat, a push service that gathers information from various sources and delivers it to the users according to their profile. Users profiles include information about subjects, types, etc. This session's invited speaker was Nicholas Belkin. After defining the "what/why/how" of User Modelling, Nick discussed its challenges and future, which can be summarised as follows: a change of focus to the user's context (goals, knowledge of situation), and more emphasis on dynamic (rather than static) models.

Answer Organisation

In this session we considered systems that present answers (typically the results of a search engine) in a customised way. Achim Leubner discussed a model for Personalized Ranking, where the ranking of answers can be influenced by (set of) term preferences. A prototype has been implemented in Java. Henry Naftulin introduced a Clustering Algorithm for the Internet and E-Commerce. In this approach, the user has full control over which attributes are used for classification or presentation.Finally, David Harper presented two systems for Information Access: SketchTrieve and WebCluster. He concluded by discussing the benefits and trade-offs between collaborative and individual customisation.

Adaptive Hypertext

In the final session, we examined systems that do not only adapt the contents (as in the first session), but also the hypertext structure. The first presentation, by Carolyn Watters, described a Medical Portal, which provides an integrated view of medical data through a Web interface.The interface may be tailored to group or personal needs (eg. patient vs physician). Richard Bodner proposed a Dynamic Hypertext System where the hypertext links are determined by the author's recommendations and the user's interaction. Their system, ClickIR, was evaluated in TREC-7. The issue of Evaluation was the also the subject of Ross Wilkinson's talk. Ross started with the traditional IR evaluation framework, precision/recall, and then discussed common problems in experiments: how to make them statistically significant,  how to recognise and control covariant variables, how to measure users, etc. He concluded by reiterating the importance of user experiments.


Discussion of Topics Arising

A recurrent theme in the workshop was the importance of dialogue in customised information delivery. Dialogue is a process through which the system can form a better understanding of the user's characteristics and information need. This can happen by explicit questioning, but also just the user's interaction with the system (for example, whenever the user clicks on a hypertext link, it is like a follow-up question). The fundamental problem of adaptive presentation can then be seen as supporting communication between user and system, using (amongst other things) a user model. In this context, the evaluation of a system consists of evaluating the effectiveness of this communication.

The discussions in the workshop focused on three problems, central to the issue of communication in a tailored environment: user modelling, adaptive presentation, and evaluation.

User Modelling

In his presentation, Nick Belkin defined user modelling as "some understanding and/or representation of salient characteristics of the user in the information system, to support the user in interaction with information (resolve the user's problem)". Amongst the things that can be modelled are the information need, the user's knowledge about the topic and the system, the user's goals, etc. The information to populate the user model can be obtained through explicit questioning, inference based on characteristics of the interaction, or evidence from external sources.

An idea emerging from our discussions about user modelling is that a user model should be simple, so that it can be shared, that is, the system can communicate it to the user, and the user can understand it. This is also referred to as  "scrutability" in the literature. This makes perfect sense in a dialogue environment: the dialogue cannot make sense unless the user and the system have a reasonable understanding of each other, and much of the dialogue in fact is spent to refine or correct that understanding. The dialogue environment also means that more emphasis should be placed on dynamic rather than static models.

An interesting question that was raised is whether there is a need in Information Retrieval to change focus from the information need of user, which is very punctual and local, towards the idea that users request information to achieve a goal, which forces a more global and potentially continuous approach. Again this fits perfectly in a dialogue environment, which usually implies the concept of a context for the information need, and temporality.

Finally, we discussed the respective benefits of individual models and collective models. The user modelling community has long made this distinction between individual and stereotypical models, where stereotypical models refer to a class of users or user community. Individual profiles can usually match users needs better, but they are more subject to variations than stereotypical models. Stereotypical models can also be less "intrusive" to the user: they generally require less information per user, as they can use information from previous users and interactions to build their model.

Adaptive Presentation

The general problem of adaptive presentation can be viewed as deciding what to say and how to say it: what information to convey/omit, how to organise information in a coherent way, how to group that material into coherent units, and how to present it. In the context of a communication process, adaptive presentation can make use of discourse and language models: it should support dialogue and take past discourse into account.

Evaluation

Traditional measures such as precision/recall are not appropriate for evaluating communication between user and system. Current user experiments tend to evaluate systems as a whole: it is difficult to isolate factors (exactly what in the system went wrong?) and interpret the results (why did it go wrong?).

A key issue in evaluation is the cost of modelling versus its benefits. While we assume that the more information we know about a user, the better customisation we can provide, there is a trade-off between cost and effectiveness. There is a communication cost for the user to supply information, and there is a higher cost for implementing a solution with a more complex user model. While user burden can partially be solved by acquiring some information automatically, this might introduce another problem: the system is not "predictable" anymore, and the user might get confused as to what is happening behind the scenes.


Conclusions

The workshop helped us appreciate the importance of communication in customised information delivery. When viewed in this context, the purpose of a user model is to inform a communication dialogue, and evaluation consists of measuring this communication.

The workshop has highlighted some of the issues and challenges for customised information delivery. Some of the main challenges for the future are:



For more information about our workshop please visit our Web page at http://www.ted.cmis.csiro.au/sigir99/, or contact any of the organisers:

Maria Milosavljevic, François Paradis, Cécile Paris and
Ross Wilkinson