New general manager of Meta’s fundamental research activity in AI, Joëlle Pineau recounts her rise within Facebook’s laboratories and deciphers the latest advances in deep learning models.
JDN. Why did Meta set up a fundamental research laboratory in artificial intelligence?
Joelle Pineau. The mission of Meta’s FAIR laboratory is to push the state of the art in artificial intelligence ever further. By having a fundamental research laboratory, Meta is in a position to face the challenges of tomorrow in terms of AI. Faced with the dazzling advances in this technological field over the last decade, FAIR relies on an open and transparent research model, including in terms of models and source codes. The objective is to stimulate feedback and the exchange of ideas on our work with our various academic and industrial partners in order to move our projects forward quickly.
Pytorch whose development we have entrusted to a foundation (the PyTorch Foundation, editor’s note)and which partly redesigned the way to develop AI, is one of our main successes.
You have just taken the helm of FAIR. What have your roles been since joining Meta in 2017?
I was initially hired to set up the FAIR lab in Montreal. Which took about two years. In the process, I was appointed co-director of FAIR alongside Antoine Bordes who is based in Paris. My role was to bring our mission to our teams of researchers, the company and the scientific community, the challenge being to make the strategy clear and the vision aligned to achieve our objectives. I had to make sure that the teams had the necessary resources to carry out their projects in terms of budget, computing capacities… But also to define and apply a data strategy so that the use we have of the data as part of our open science approach is in line with legislation. Finally, my objective was to bring into our teams the best researchers in the cutting-edge fields of AI.
“We have extended in particular to reinforcement learning, robotics or even responsible AI”
In five years, we have evolved a lot. Historically, FAIR was primarily focused on deep learning. At the time, we only had about fifty researchers. It is therefore in this area that we had to position ourselves. Over the years, our membership has grown. This allows us today to cover a much wider range of areas. We have extended in particular to reinforcement learning, robotics and even responsible AI.
Have you continued to follow certain research topics closely?
I continued to conduct research in interactive dialogue systems and speech processing through language models and reinforcement learning. This is the theoretical core of my areas of historical research. I also work a lot on AI in healthcare. This is a process that I began long before joining Meta, when I was a professor at McGill University in Montreal, and which continues today. I am still supporting two post-docs on this subject.
You also worked on Meta’s BlenderBot conversational AI. How did you intervene?
A whole team of FAIR researchers carried out the scientific aspect of the project. We decided to deploy BlenderBot on the Internet in the United States. I worked a lot on this last step. The main goal was to make sure the bot behaved reasonably. A strategy has been defined to assess the performance of dialogue quality and security in order to achieve a trusted AI.
Ultimately, this deployment aimed to provide access to this technology, both to communities of researchers and to the general public. It’s a way to popularize our work, but also to understand user behavior. This experience allows us to collect usage data that we obviously do not have in the laboratory and which allows us to improve our research.
Microsoft has developed a large language model with 530 billion parameters. What do you think of the race for these giant AIs in which Meta is also participating?
The community has been wondering about the relevance of these models for ten years. We all tell ourselves that their size must decrease. There are three ingredients for these AIs to succeed. Parameter volume is one of them. The second refers to the number of GPUs available to train them. We’ve gone from a few dozen to a few hundred, and now to a few thousand GPUs running for several months. Third element, it is necessary to ingest data in sufficient quantity. If we do not have a good balance between the volume of data, the computing capacity and the size of the model, the sauce does not take.
“We have enough memory to absorb even larger model sizes”
It is not yet known which of these parameters will be the first to reach its limit. We have enough memory to accommodate even larger model sizes. On the training data side, on the other hand, the room for maneuver is narrowing. This is explained by the strengthening of copyright and privacy rules, as well as by the difficulty of having access to massive quantities of information representative of a population.
What will be the escape?
The future could go through the emergence of multimode transformers capable of feeding on heterogeneous data: images, videos, language, sounds, medical data… The other major field of research concerns generative models. In this area, we have just announced the Make-A-Video technology which produces short videos from texts describing a moving scene, in the same logic as Dall-E on the still image front.
Generative AIs will soon be able to produce different data formats, for example 3D virtual reality with intelligent avatars, sound, touch and smell sensory experiences, especially in the context of the metaverse…
Joëlle Pineau was appointed on October 20, 2022 as Chief Executive Officer of FAIR, Meta’s fundamental research activity in artificial intelligence. She previously co-led FAIR with Antoine Bordes, who will now focus on leading FAIR EMEA Labs. Joëlle Pineau is an associate professor at the School of Computer Science at McGill University.