Customised Information Delivery: A SIGIR 99 Workshop
Introduction
Sessions
Adaptive Presentation
User Profile Information Systems
Answer Organisation
Adaptive Hypertext
Discussion of Topics Arising
User Modelling
Adaptive Generation
Evaluation
Conclusions
Contacts
Introduction
A major strength of electronic information is the fact that as opposed
to its paper counterpart, it can be dynamically modified and tailored to
a user's need. The problem has been tackled from different angles by different
communities. The IR community has revisited its traditional model of query/answers,
and now allows for user interaction, synthesized answers, contextual information
when presenting answers, etc. The language technology community has designed
user models and natural language interfaces to allow better communication
and understanding between the system and the user. The Web community has
taken a pragmatic approach, with scripts and virtual documents to produce
Web pages.
The aim of this workshop was to bring together these communities, in
the hope of fostering collaboration and exchange of ideas. Indeed, the
45 participants to our workshop came from different disciplines: Information
Retrieval, Filtering, Natural Language Generation, Human-Computer Interaction,
Information Science, and the Web community.
Sessions
The workshop was divided in four sessions: Adaptive Presentation, User
Profile Information Systems, Answer Organisation and Adaptive Hypertext.
Each session comprised two papers and an "invited talk". There was 30 minutes
for discussion at the end of each session.
Adaptive Presentation
In this session we discussed systems that focus on adapting the content
to the user. Caroline Eastman introduced the concept of Customizable
Information Components, which are basic chunks of information that
can be derived and combined to form documents. An hypercard implementation
and user experiments were presented. John Tait presented SYSTAR, a Syntactic
Simplification System for Aphasic Readers. Their proposal is to help
aphasic readers (a reading disorder) by simplifying the original text,
for example resolving anaphora. In her position paper, Cécile Paris
summarised Adaptive
Presentation as a communication problem between the user and the system,
and then explained how natural language generation and user modelling can
be used to achieve a presentation that "make sense" to the users in their
interaction with the system.
User Profile Information Systems
The next session was about user modelling. Two systems that make
use of a user profile were presented. Eric Lease Morgan presented MyLibrary@NCState,
a digital library that can be customised by users. The system allows users
to directly add their own links to the pages, and can present different
information according to the users' discipline. Ricardo Mazza presented
SwissCat,
a push service that gathers information from various sources and delivers
it to the users according to their profile. Users profiles include information
about subjects, types, etc. This session's invited speaker was Nicholas
Belkin. After defining the "what/why/how" of User
Modelling, Nick discussed its challenges and future, which can be summarised
as follows: a change of focus to the user's context (goals, knowledge of
situation), and more emphasis on dynamic (rather than static) models.
Answer Organisation
In this session we considered systems that present answers (typically
the results of a search engine) in a customised way. Achim Leubner discussed
a model for Personalized
Ranking, where the ranking of answers can be influenced by (set of)
term preferences. A prototype has been implemented in Java. Henry Naftulin
introduced a Clustering
Algorithm for the Internet and E-Commerce. In this approach, the user
has full control over which attributes are used for classification or presentation.Finally,
David Harper presented two systems for Information
Access: SketchTrieve and WebCluster. He concluded by discussing the
benefits and trade-offs between collaborative and individual customisation.
Adaptive Hypertext
In the final session, we examined systems that do not only adapt the contents
(as in the first session), but also the hypertext structure. The
first presentation, by Carolyn Watters, described a Medical
Portal, which provides an integrated view of medical data through a
Web interface.The interface may be tailored to group or personal needs
(eg. patient vs physician). Richard Bodner proposed a Dynamic
Hypertext System where the hypertext links are determined by the author's
recommendations and the user's interaction. Their system, ClickIR, was
evaluated in TREC-7. The issue of Evaluation
was the also the subject of Ross Wilkinson's talk. Ross started with the
traditional IR evaluation framework, precision/recall, and then
discussed common problems in experiments: how to make them statistically
significant, how to recognise and control covariant variables, how
to measure users, etc. He concluded by reiterating the importance of user
experiments.
Discussion of Topics Arising
A recurrent theme in the workshop was the importance of dialogue in customised
information delivery. Dialogue is a process through which the system can
form a better understanding of the user's characteristics and information
need. This can happen by explicit questioning, but also just the user's
interaction with the system (for example, whenever the user clicks on a
hypertext link, it is like a follow-up question). The fundamental problem
of adaptive presentation can then be seen as supporting communication between
user and system, using (amongst other things) a user model. In this
context, the evaluation of a system consists of evaluating the effectiveness
of this communication.
The discussions in the workshop focused on three problems, central to
the issue of communication in a tailored environment: user modelling, adaptive
presentation, and evaluation.
User Modelling
In his presentation, Nick Belkin defined user modelling as "some understanding
and/or representation of salient characteristics of the user in the information
system, to support the user in interaction with information (resolve the
user's problem)". Amongst the things that can be modelled are the information
need, the user's knowledge about the topic and the system, the user's goals,
etc. The information to populate the user model can be obtained through
explicit questioning, inference based on characteristics of the interaction,
or evidence from external sources.
An idea emerging from our discussions about user modelling is that a
user model should be simple, so that it can be shared, that is, the
system can communicate it to the user, and the user can understand it.
This is also referred to as "scrutability" in the literature. This
makes perfect sense in a dialogue environment: the dialogue cannot make
sense unless the user and the system have a reasonable understanding of
each other, and much of the dialogue in fact is spent to refine or correct
that understanding. The dialogue environment also means that more emphasis
should be placed on dynamic rather than static models.
An interesting question that was raised is whether there is a need in
Information Retrieval to change focus from the information need of user,
which is very punctual and local, towards the idea that users request information
to achieve a goal, which forces a more global and potentially continuous
approach. Again this fits perfectly in a dialogue environment, which usually
implies the concept of a context for the information need, and temporality.
Finally, we discussed the respective benefits of individual models and
collective models. The user modelling community has long made this distinction
between individual and stereotypical models, where stereotypical models
refer to a class of users or user community. Individual profiles can usually
match users needs better, but they are more subject to variations than
stereotypical models. Stereotypical models can also be less "intrusive"
to the user: they generally require less information per user, as they
can use information from previous users and interactions to build their
model.
Adaptive Presentation
The general problem of adaptive presentation can be viewed as deciding
what to say and how to say it: what information to convey/omit, how to
organise information in a coherent way, how to group that material into
coherent units, and how to present it. In the context of a communication
process, adaptive presentation can make use of discourse and language models:
it should support dialogue and take past discourse
into account.
Evaluation
Traditional measures such as precision/recall are not appropriate for evaluating
communication between user and system. Current user experiments tend to
evaluate systems as a whole: it is difficult to isolate factors (exactly
what in the system went wrong?) and interpret the results (why did it go
wrong?).
A key issue in evaluation is the cost of modelling versus its benefits.
While we assume that the more information we know about a user, the better
customisation we can provide, there is a trade-off between cost and effectiveness.
There is a communication cost for the user to supply information, and there
is a higher cost for implementing a solution with a more complex user model.
While user burden can partially be solved by acquiring some information
automatically, this might introduce another problem: the system is not
"predictable" anymore, and the user might get confused as to what is happening
behind the scenes.
Conclusions
The workshop helped us appreciate the importance of communication in customised
information delivery. When viewed in this context, the purpose of a
user model is to inform a communication dialogue, and evaluation
consists of measuring this communication.
The workshop has highlighted some of the issues and challenges for customised
information delivery. Some of the main challenges for the future are:
-
Understand the problem better. Because the problem of customisation
spans across several disciplines, no single discipline can have a complete
"picture" of it. For multi-disciplinary collaboration to happen, it is
necessary to have a spirit of "openness" and to overcome the differences
in philosophy and terminology.
-
Come up with a framework for evaluation. Evaluating one system is
hard enough, but we must also be able to compare different systems which
may have different tasks and apply to different domains.
-
Know when to use customisation. Know in which situations we need
customising, and which level of customising is appropriate.
For more information about our workshop please
visit our Web page at http://www.ted.cmis.csiro.au/sigir99/,
or contact any of the organisers:
Maria Milosavljevic,
François Paradis,
Cécile Paris and
Ross Wilkinson