Abstract: We have seen astonishing progress of machine learning research in the last years. Unfortunately, it is often difficult to translate this academic progress into deployable applications, due to the constraints and challenges imposed by production settings. In this talk, I will present some of my recent research in the area of data management for machine learning, which tackles these problems. Furthermore, I will put a special focus on three challenges that I see for building industrial-scale recommender systems. In particular, I will outline ideas on how to scale to datasets with billions of interactions, understand the impact of response latency on the performance of a deployed recommender system, and make the "right to be forgotten" a first-class citizen in real-world ML systems.

Abstract: We address the problem of customer retention (churn) in applications installed on over the top (OTT) streaming devices. In the first part of our work, we analyze various behavioral characteristics of users that drive application usage. By examining a variety of statistical measures, we answer the following questions: (1) how do users allocate time across various applications?, (2) how consistently do users engage with their devices? and (3) how likely are dormant users liable to becoming active again? In the second part, we leverage these insights to design interpretable churn prediction models that learn the latent characteristics of users by prioritizing the specifications of the users. Specifically, we propose the following models: (1) Attention LSTM (ALSTM), where churn prediction is done using a single level of attention by weighting on individual time frames (temporal-level attention) and (2) Neural Churn Prediction Model (NCPM), a more comprehensive model that uses two levels of attentions, one for measuring the temporality of each feature and another to measure the influence across features (feature-level attention). Using a series of experiments, we show that our models provide good churn prediction accuracy with interpretable reasoning. We believe that the data analysis, feature engineering and modeling techniques presented in this work can help organizations better understand the reason behind user churn on OTT devices.

Abstract: Classical approaches to recommendation systems like collaborative filtering learn a static model given the user historic interaction data. These approaches do not perform well in dynamic environments where the sets of users and items are continually changing. Users convey their preferences implicitly by providing feedback in the form of clicks, views and ratings, as they interact with the system. Utilizing this feedback in an online manner is crucial for building a good user experience. Contextual bandit algorithms provide a suitable framework for learning user preferences online by balancing the explore-exploit trade-off. Much of the bandit literature focuses on choosing one item, we extend these algorithms to recommend a list of actions by assuming a cascade click model. We provide an empirical study across different scenarios to showcase the benefits of collaborative online learning and exploration. Finally, we propose a novel algorithm - Collaborative Pairwise Ranking (CPR), that uses pairwise differentiable gradient descent to perform online ranking collaboratively. We showcase that this approach outperforms state-of-the-art collaborative bandit approaches, especially in the presence of noisy feedback common in practical scenarios.

Abstract: In the actual context of the media crisis, online media companies need cost-effective technological solutions to stay competitive against huge monopolistic software companies massively feeding content to users. News recommender systems are well-suited solutions, even if current commercial solutions are well above most online media's budget. In this paper, we present a case study of our deployed deep news recommender system at Le Devoir, an independent french Canadian newspaper in the province of Quebec. We expose the software architecture and the issues we have met with their solutions. Furthermore, we present four qualitative and quantitative analyses done with our custom monitoring dashboard: offline performances of our models, embedding space analysis, fake-user testing and high-traffic simulations. For a tiny fraction of the available commercial solutions' prices, our current simple software architecture based on the Docker, the Kubernetes and open-source technologies in the cloud has demonstrated to be easily maintainable, scalable, and cost-effective. It also shows excellent offline performance and generates high-quality embeddings as well as relevant recommendations.

Abstract: Cross-Domain Recommendation (CDR) is an important task in recommender systems. Information can be transferred from other domains to target domain to boost its performance and relieve the sparsity issue. Most of the previous work is single-target CDR (STCDR), and some researchers recently propose to study dual-target CDR (DTCDR). However, there are several limitations. These works tend to capture pair-wise relations between domains. They will need to learn much more relations if they are extended to multi-target CDR (MTCDR). Besides, previous CDR works prefer relieving the sparsity issue by extra information or overlapping users. This leads to a lot of pre-operations, such as feature-engineering and finding common users. In this work, we propose a heterogeneous graph framework for MTCDR (HeroGRAPH). First, we construct a shared graph by collecting users and items from multiple domains. This can obtain cross-domain information for each domain by modeling the graph only once, without any relation modeling. Second, we relieve the sparsity by aggregating neighbors from multiple domains for a user or an item. Then, we devise a recurrent attention to model heterogeneous neighbors for each node. This recurrent structure can help iteratively refine the process of selecting important neighbors. Experiments on real-world datasets show that HeroGRAPH can effectively transfer information between domains and alleviate the sparsity issue.

Abstract: Recommender systems aim to recommend items to users based on their interests. Traditional models usually adopt batch processing. Considering that user feedback is generated continuously, it becomes desirable to design models that are capable of learning as data arrives. In this work, we propose an incremental graph of sequential user interactions using implicit feedback from a data stream, with the assumption that user behavior can be extracted from such sequence of interactions as time passes. The model was evaluated by recommending items with different strategies, and such strategies were compared with an incremental matrix factorization algorithm, using a prequential approach. Results highlight the potential of the proposed method, which obtained superior accuracy than the baseline with generally better update and recommendation times.

Abstract: Online Accommodations Platforms match guests searching for accommodation with hospitality service providers. A fundamental characteristic of efficient platforms is the ability to satisfy the needs and preferences of the guests. To achieve this goal, a common search tool is the Results Filtering capability which allows users to refine query results with explicit criteria. However, as supply grows and diversifies, more filtering options become available, reaching hundreds of different criteria for one query, and making it hard for customers to find the ones that are relevant to them. In this work we present the implementation of an Accommodation Filters Recommender System addressing this issue. The problem poses several challenges around recommendations feedback, user experience constraints, and non stationarity among others. We provide an end-to-end description of the System, discuss implementation issues and provide techniques to address them including a large scale distributed online learning architecture. The solution was validated through several Online Controlled Experiments performed in Booking.com, a top Online Travel Agency serving millions of daily users, showing statistically significant results on various user behaviour metrics indicating a strong positive effect on User Engagement.

Abstract: Mobile gaming has become increasingly popular due to the growing usage of smartphones in day to day life. In recent years, this advancement has led to an interest in the application of in-game recommendation systems. However, the in-game recommendation is more challenging than common recommendation scenarios, such as e-commerce, for a number of reasons: (1) the player behavior and context change at a fast pace, (2) only a few items (few-shot) can be exposed, and (3) with an existing hand-crafted heuristic recommendation, performing randomized explorations to collect data is not a business choice that is preferred by game stakeholders. To that end, we propose an end-to-end model called DFSNet (Debiasing Few-Shot Network) that enables training an in-game recommender on an imbalanced dataset that is biased by the existing heuristic policy. We experimentally evaluate the performance of DFSNet both in an offline setup on a validation dataset and online in a real-time serving environment, illustrating the correctness and effectiveness of the trained model.

Abstract: Watch time has been a subject of interest for recommender systems in recent years. Music, podcast and video recommendations based on or amplified by consumption time optimization are often employed to boost perceived user satisfaction, subscriptions, engagement or to decrease number of bounces. Finding a fragile balance between several different metrics describing users' behaviour might be challenging, especially in the area of video recommendation. In this paper, we design online algorithms for modelling relationship between click probability and expected watch time on a video. We explore means of combining and balancing click-based and watch time optimizing models in an online multi-criteria setting. We present experiments that involve watch time, CTR and watch ratio. Furthermore, the paper describes empirical evaluations on live traffic and illustrates that our approach has succeeded in outperforming a non-trivial baseline in a controlled manner.