Economics provides an intuitive and natural way to formally represent the costs and benefits of interacting with applications, interfaces and devices. By using economic models it is possible to reason about interaction, make predictions about how changes to the system will affect performance and behavior, and measure the performance of people’s interactions with the system.

Development of the majority of the leading web services and software products today is generally guided by data-driven decisions based on evaluation that ensures a steady stream of updates, both in terms of quality and quantity. Large internet companies use online evaluation on a day-to-day basis and at a large scale. The number of smaller companies using A/B testing in their development cycle is also growing. Web development across the board strongly depends on quality of experimentation platforms. In this tutorial, we will overview state-of-the-art methods underlying everyday evaluation pipelines at some of the leading internet companies.

We invite software engineers, designers, analysts, service or product managers — beginners, advanced specialists, and researchers — to join us at the conference SIGIR 2019, which will take place in Paris from 21 to 25 of July, to learn how to make web service development data-driven and do it effectively.

This is the third version of the tutorial that have already been presented at WWW and KDD, where it was one of the most popular. We present you a program of a balanced mix between an overview of academic achievements in the field of online evaluation and a portion of unique industrial practical experience shared by both the leading researchers and engineers from Yandex and Facebook. Whether you work at a company, might do so in the future or plan to drive the practice of online evaluation in academia, we welcome you at our tutorial. Please, visit the web-site with materials from the previous versions of our tutorials by the link above for more information.

This tutorial aims to weave together diverse strands of modern learning-to-rank (LtR) research, and present them in a unified full-day tutorial. First, we will introduce the fundamentals of LtR, and an overview of its various subfields. Then, we will discuss some recent advances in gradient boosting methods such as LambdaMART by focusing on their efficiency/effectiveness trade-offs and optimizations. We will then present TF-Ranking, a new open source TensorFlow package for neural LtR models, and how it can be used for modeling sparse textual features. We will conclude the tutorial by covering unbiased LtR – a new research field aiming at learning from biased implicit user feedback.

The tutorial will consist of three two-hour sessions, each focusing on one of the topics described above. It will provide a mix of theoretical and hands-on sessions, and should benefit both academics interested in learning more about the current state-of-the-art in LtR, as well as practitioners who want to use LtR techniques in their applications.