“Contrary to what many feared, European works do not disappear from the Netflix homepage, even when the user chooses not to watch any European titles”
Industry Report: Distribution, Exhibition and Streaming
Grégoire Bideau and Steven Tallec • Researchers, Chaire PcEn, Université Paris 1 Panthéon-Sorbonne
The two scholars recently conducted an experiment during which 20 bots watched Netflix every day for eight days and tried to gain insights into European works’ prominence on the platform
We spoke to Grégoire Bideau and Steven Tallec, researchers at Chaire PcEn. The two scholars recently published a study titled Algorithmic Panic: European Contents on the Netflix Homepage, during which 20 bots watched Netflix every day for eight days and tried to gain insights into European works’ prominence on the platform.
Cineuropa: Why did you start studying Netflix’s algorithm?
Steven Tallec: For the past couple of years, we’ve had the opportunity to speak to a number of stakeholders in the European audiovisual industry, from regulators to producers, government officials and analysts. During our conversations, the question of the impact of Netflix’s recommender system on the visibility of European works came up almost every time, but there did not seem to be any reliable data from which to determine whether that impact was positive or negative, although the latter was often assumed.
Grégoire Bideau: Prominence is an important aspect of the EU’s AVMS Directive, which states that services “shall ensure prominence of [European] works”, and not simply set aside a 30% share of their catalogues for them. Even though “prominence” has never been properly defined, let alone regulated, professionals do seem aware of its importance in the circulation of European works. However, it is a slippery notion in an algorithmically personalised context, where prominence is tied to past viewing behaviour. As such, it is very difficult to study quantitatively, even though it plays an important mediating role between a vast catalogue of contents and the end user [read more]. This is why, at the Chaire PcEn, we placed content prominence at the heart of our research programme two years ago. We chose Netflix as our first test subject because of its popularity and its famed recommender system, and we hoped that our research would shed light on an otherwise opaque mechanism.
What was your methodology when conducting this experiment?
ST: First, we gathered metadata for each title available on Netflix in France by finding its corresponding page on IMDb (not by hand, but with a specially designed script), which allowed us to collect its producing countries, languages, actors and so on.
GB: In parallel, we set up a cluster of 20 tiny computers (Raspberry Pi’s), each tasked with creating its own Netflix profile on a dedicated account [read more]. Then, each computer was fed a schedule of when to watch Netflix (in this case, three hours each day for eight days) and a list of titles to watch. Ten computers were given a list made up solely of European titles, and the remaining ten were given a list of non-European works. All lists were arranged in a random order so as to prevent any selection bias and to ensure that all computers were watching different titles at the same time. We relied on automated web browsing so that there was no human intervention during the whole of the experiment. We collected the contents of each bot’s homepage once a day, including the exact placement of each title.
Could you talk us through the results? What kind of insights did you gain in terms of the circulation of European works?
GB: Contrary to what many feared, European works certainly do not disappear from the Netflix homepage, even when the user chooses not to watch any European titles. There is a slight downward trend in the number of thumbnails promoting European works, but nothing that would make European content absent from the homepage. As for the bots that watched only European content, the number of European thumbnails on the homepage increases, but not so much as to create a “filter bubble” that would suggest only European titles to the user. In both cases, it seems that the extent to which the recommender system influences the visibility of European works has been greatly overestimated.
ST: Of course, this also has to do with using “European” as the sole differentiating variable between our two sets of bots. European works do not constitute a genre, nor are they perceived as particularly similar to all of Netflix’s users. In fact, we could go as far as to say that the diversity of European works partly ensures their prominence on Netflix, as they are likely to be included in a wide variety of taste clusters, and thus recommended to users with very different tastes.
This eight-day experiment is certainly interesting, but perhaps a longer run would even provide more solid, additional takeaways. Are you willing to prolong it, or if not, why do you believe such a length is appropriate?
GB: Sure, a longer experiment would be interesting, and we hope to be able to carry out other experiments that will last longer in the future (technical issues notwithstanding!). We felt that eight days, or just over a week, was an appropriate period for looking at European works because any longer would have meant some extremely unusual behaviour from our bots, which could have skewed the results.
What will your next steps be? Would you be interested in conducting a similar study on other VoD platforms?
ST: First, we will continue to put this methodology to use in future experiments, where we will explore other variables, such as languages and genres. We will also strive to look more closely at the results, and provide more in-depth analyses of the movements on the homepage. We have already started a partnership with Professor Seok-Kyeong Hong at the Seoul National University to understand how Netflix’s recommender system promotes Korean content in France. The results will be presented at a conference in Seoul on 9 April. It would also be interesting to measure the prominence of European works in English-speaking countries, such as the UK and the USA, but that would mean either moving our computers to another country to access its Netflix catalogue or using a VPN for the whole process.
GB: Of course, it is technically feasible to conduct a similar experiment on other VoD platforms that use algorithmic personalisation. Some elements of the methodology would have to be adapted, of course, requiring quite a bit of effort to automate browsing and data collection on those platforms, but the crux of the process would remain the same. Amazon Prime Video, Disney+ or even Twitch would be interesting to look at, for example.
Did you enjoy reading this article? Please subscribe to our newsletter to receive more stories like this directly in your inbox.