Spotify's Research Director on AI and the Personalization of MusicSpotify's Research Director on AI and the Personalization of Music

Spotify recently unveiled an AI-powered personal DJ tool that learns about the listener’s tastes - and offers a personalized, curated lineup of music. Partly powered by tech from OpenAI, the DJ also comes with commentary around tracks and artists.

Currently in beta, the DJ tool is part of Spotify’s growing focus on personalization, according to its senior director of research, Mounia Lalmas-Roelleke, who joined the AI Business podcast.

She teased Spotify’s generative AI plans and explained how reinforcement learning powers its user personalization experiences. She also offered expert advice on dealing with research setbacks.

The following is an edited transcript of that conversation. You can listen to the full chat in the latest episode of the AI Business Podcast below, or wherever you get your podcasts.

AI Business: Talk to us a bit about your work at Spotify.

Mounia Lalmas-Roelleke: I lead a team of around 35 research scientists. Our focus is really on personalization and building algorithms that power personalization on Spotify.

The mission of Spotify is to unlock the potential of human creativity by giving a million creative artists the opportunity to live off their art, and billions (of listeners) have found the opportunity to enjoy and be inspired by it. What my team is doing … is to help support Spotify’s mission by creating a personalized experience that fulfills and delights Spotify artists and listeners.

We are able to suggest to users other kinds of content they are likely to like and enable them to discover both new music and podcasts they have never heard before. It is providing the best possible experience.

AI Business: A big thing about Spotify is personalization – it is all about song and playlist recommendations. How does that tie into your work?

Lalmas-Roelleke: When you look at personalization, recommendation and search, there is the algorithm, but there are a lot of things all around it. As researchers, we bring in-depth expertise to approach personalization from various angles. That expertise includes human-computer interaction, language technology, machine learning, algorithmic responsibility, evaluation, search and recommendation, and also user modeling.

What we are trying to do with our various expertise is to understand the user, understand their behavior, understand the artists, their content and how often this is constantly changing from day to day so we can constantly create unique, and the best experience fitting user needs.

Another part of personalized recommendations is to introduce users to new artists as it is not just the user, there is also the creator path.

AI Business: Your team is looking at utilizing a reinforcement learning-based approach to your recommendation tools. How does this approach yield results?

Lalmas-Roelleke: Reinforcement learning allows us to think about the bigger picture. In the context of consuming music, podcasts and so on, we should view the user on a journey − what a user likes now is not what they are going to like tomorrow, in a month, or in a year's time.

The way a standard recommendation tool works is by predicting the next click independently on the journey. What we are trying to do with reinforcement learning is to try to predict the next and next and next … and the future that comes with it.

We try to understand where the user is on this journey, what could be the next journey, and so on. Reinforcement learning is trying to answer how satisfied you are as well as trying to nudge a user towards content that they may not have heard before, but is still relevant, and really helping them on the journey.

AI Business: Could your CX work potentially be applied to other businesses or markets, especially given the increased demand from consumers for more personalized commerce experiences?

Lalmas-Roelleke: Recommendation systems and recommendation algorithms are everywhere; they are not just specific to Spotify. For example, as researchers, when we attend conferences that are dedicated to machine learning, human-computer interaction, natural language, and so on, we have many papers that are about recommendations. The recommendation landscape is big.

The algorithms that people are developing, we included, they are not so dramatically different. They tend to not change dramatically. It is about putting it all together, that makes a big difference. The same algorithm may work for other industries, but they will have to be adapted to their use case, to their domain and to the expectations of the users.

AI Business: A big push by Spotify, and others, is talk content – things like podcasts and audiobooks. How do they factor into your work and the recommendation systems you help lay the groundwork for?

Lalmas-Roelleke: How users consume this type of content is very different. You do not listen to music the way you read news, so the experience itself and the expectation of the user is going to be super important, then the data, the interaction between the user and the content and so on. A big part is how to interpret interactions.

Podcasts are still audio content, but it is a different type of content. It is not easy to find the right podcasts; there is a big time commitment. When a user decides to go into podcasts, they pick them very carefully. Podcasts are long-form and require time, dedication and willingness to explore. And in that sense, even if it is the same app, it is different audio for the same user.

AI Business: On the subject of talk content, rival Apple recently published a series of audiobooks narrated by AI – something that stems from the hype around generative AI. Is this something your team is looking at right now and if so, how long can we expect before AI-generated content appears on Spotify?

Lalmas-Roelleke: I am afraid I cannot reveal too much about what Spotify may be doing in this space. But it is undeniable that generative AI is a topic that is just crazy. It is generating a lot of interest right now. ... But of course, we are paying a lot of attention to developments in this space.

What I can share is that we are already scaling up our ability to transcribe and model much of our podcast catalog with large language technology. And we are optimistic that this will provide a major advance in two particular areas: improving our ability to personalize and recommend podcast content because we get a better understanding of the content, and also improving our ability to monitor and moderate unsafe content.

More generally, we are also building a prototype using a large language model that provides our users with novel ways to navigate music, podcasts and audiobook catalogs. … Stay tuned.

AI Business: In terms of your research, what are some of the difficulties you’ve had to overcome to get your ideas over the line?

Lalmas-Roelleke: Spotify strongly believes in its research mission. It is important to understand how we view research; it is about solving problems that have no known or obvious solution. And there are going to be a lot of those. That is why we have researchers at Spotify. And the thing that I always say to people is if we knew it would work, it would not be called research. That is why we are part of the product cycle.

(That said), researchers have a lot of in-depth expertise and usually have a good intuition of what could work − and it is important to understand what could work.

Not all research is going to succeed and go into production, but that is normal. If everything was 100%, it will not be research. And even if something does not go into production, there are a lot of lessons learned and often provides a guide for future strategies and so on. In a way, we feel quite good in terms of those difficulties. Communication is crucial on day one.

AI Business: Finally, can you provide our listeners with some advice in terms of approaching research projects: things to consider, approaches to adhere to and how to react when things do not go the way they expect?

Lalmas-Roelleke: Make sure that the researchers are part of a (line-of-business) team. What we actually do at Spotify is we have researchers embedding with teams, especially when they start so they understand the problem, build a relationship with a team and build a baseline. This baseline is so important because they then understand the problem better. And then we can do version 1 and version 2, and so on. It also improves communication.

Understand that when there is research happening, it may not lead to something, but we know what you are doing. It is not just about the researcher; it is about building this community that is trying to solve a problem together.