Senior Researcher · CNRS, ISIR, Sorbonne Université
I am a senior researcher within the CNRS in France, working in the MLIA team of the Institute of Intelligent Systems and Robotics (ISIR) located in Sorbonne Université.
My research is on information access — how to represent text and structured data so machines can retrieve, reason over, and generate it — at the interface of information retrieval and computational linguistics.
Efficient and effective neural retrieval. Much of my recent work develops learned sparse retrieval, where queries and documents are represented in the vocabulary space: this keeps the interpretability and inverted-index efficiency of bag-of-words models while reaching the effectiveness of dense ones (SPLADE, SAE-SPLADE). A companion line makes strong rankers cheap enough to deploy, through principled token pruning of late-interaction models (lossless pruning, a Voronoi-cell view) and minimal-interaction cross-encoders (MICE).
Understanding what neural models actually do. I increasingly work on opening Neural models up — how cross-encoders or late interaction models match terms (white-box ColBERT, matching mechanisms, which neurons matter, IR Lens) — and how large language models store and arbitrate knowledge (parametric vs. contextual, entropy neurons), represent entities (entity representations, ToMMeR) and positions (Give it Space!).
Generative access and structured reasoning. I study conversational and generative retrieval — contextualizing sparse models for dialogue (CoSPLADE) and generating keyword queries with reinforcement learning (QueStER, STORM) — and reasoning over tables (relational decomposition, structural encoding).
Representations across languages and modalities — and how to evaluate them. Other threads learn multilingual and multimodal representations (MEXMA, Mixture of Languages) and understand visually-rich documents (LoRaLay, Skim-Attention), alongside reference-less evaluation of generation (Data-QuestEval) and performance prediction for neural retrieval (are we there yet?).
This work provides evidence that structured entity representations exist in early transformer layers and can be efficiently recovered with minimal parameters, and reveals that diverse architectures converge on similar mention boundaries, confirming that mention detection emerges naturally from language modeling
SPLADE models can ditch their token-based vocabularies for a latent semantic space learned by Sparse Auto-Encoders, maintaining retrieval performance while boosting efficiency.
A global view on the sparse neural IR model SPLADE, summarizing and extending our papers so far
This paper describes an Information Retrieval framework library I have been developing since 2019. It is now a mature library, and is used by my two PhD students to run and design experiments on new IR models.
Recent strides in the development of generalist embodied agents have yielded remarkable achievements in addressing complex challenges across various domains, including locomotion, navigation, and manipulation within robotics. These advancements showcase a growing capacity for intelligent systems to interact with and understand their physical environments. However, a significant portion of current approaches (e.g., visual-language-action models and behavioral foundation models) predominantly adhere to the 'pre-training and then deploying' paradigm, a methodology that gained considerable popularity through large language models (LLMs). This framework typically involves training a single, 'flat' policy on extensive datasets, which is then deployed to tackle a diverse array of tasks at inference time through prompting. While effective for a wide range of applications, this approach often falls short in fostering deeper cognitive abilities. A key limitation of these models is their inherent lack of explicit capabilities for temporal abstraction and hierarchical reasoning. These two capabilities are widely regarded as fundamental for solving complex, long-term embodied tasks. Temporal abstraction allows agents to understand and plan over extended periods, recognizing the sequence and duration of actions and their outcomes. Hierarchical reasoning, on the other hand, enables agents to break down complex goals into a series of manageable sub-goals, fostering a structured approach to problem-solving. Without these capabilities, agents may struggle with tasks requiring multi-step planning, adaptation to novel situations, or understanding the broader effect of their actions. The proposed PhD research will specifically address these identified gaps. The candidate will embark on a focused exploration of how to augment existing techniques for generalist embodied agents. The central objective is to integrate functionalities that allow these agents to represent and progressively acquire temporal and hierarchical structures. This enhanced capability is crucial for enabling agents to efficiently explore, plan, and learn within complex, open-ended environments. By developing models that can intrinsically build and leverage these structures, the research aims to pave the way for more sophisticated and adaptable embodied intelligence, capable of tackling real-world challenges with greater autonomy and understanding.
The project aims to rethink recommender systems for massive, dynamic catalogs by leveraging generative models and continuous representations instead of discrete identifiers. The goal is to improve generalization to rare or new items, better capture the evolution of user intent, and unify search and recommendation within a single engine. The main directions are: - Semantic representations: directly predict item embeddings using generative models (diffusion or autoregressive), then retrieve products via similarity. - Multi-scale architecture: combine a lightweight module that captures recent intent with a more robust model representing long-term preferences. - Search/recommendation unification: train a shared generative engine on queries, histories, and interactions, combining collaborative signals with product features. - Context adaptation: model shifts in user intent through latent variables or generative conditioning. Expected outcomes include a scalable, streaming system capable of recommending new items, adapting to context drift, and powering LLM-based conversational interfaces.
The project titled ''Improving Emotion Detection in Mental Health Patients' Medical Records Using Deep Learning'' aims to enhance emotion detection in clinical notes using Transformer-based Large Language Models (LLMs). The focus is on accurately interpreting complex emotional cues and extracting psychiatric stressors to provide a comprehensive understanding of mental health states. Objectives include developing a novel LLM-based system for emotion detection, optimizing its environmental sustainability, and testing the system with mental health professionals to ensure its practical utility and effectiveness in real-world settings. This interdisciplinary project seeks to contribute to more accurate and nuanced tools for mental health diagnosis and early intervention.
The high potential of language models (LMs) is accompanied by risks associated with the generated output, which may represent content that is not aligned with user preferences or may contain information that is not grounded in reality (i.e. hallucination). Many studies have proposed methods for mitigating these risks and controlling the output of LMs (e.g. reinforcement learning or retrieval augmented generation), but this area remains under-explored. To better control the generation process, we propose to define a semantic compatibility between a generated text and a repository (i.e. a database and/or user queries). We wish to explore the extent to which this notion of semantic compatibility can be defined for restricted or more generic scopes, if necessary via statistical modeling, and exploited to modify the internal workings of generative models or to evaluate them a posteriori. In this context, we plan to study the relationship between semantic compatibility and semantic similarity. Still with the aim of producing more reliable generative models, we also plan to follow in the wake of studies into the localization of information content within Transformer-type architectures. In this sense, we propose to verify the extent to which an operation on model weights aimed at enriching the information content a posteriori can be carried out successfully. We also intend to carry out a systematic study of the differential impact - in terms of performance on a variety of language processing tasks - of technical choices in the architecture and overall development strategy of language models.
Natural language processing tools using large generative language models have excelled in complex tasks, becoming integral in digital environments for information access, analysis, reformulation, and content generation. As their adoption grows, evaluating these models' real-world performance, risks, and limitations, particularly their ''social biases,'' is crucial. Social biases manifest as preferences for or against certain social groups, seen in quantitative representation or qualitative depiction, often perpetuating stereotypes or inaccuracies. Bias evaluation is now a key aspect of model assessment. Research focuses on identifying, quantifying, and mitigating biases in LLMs, enhancing model transparency, and summarizing political debates, a field related to multi-document summarization. Effective summarization involves handling text heterogeneity, ensuring unbiased representation, and maintaining transparency. The proposed research aims to develop models blending information retrieval with generative models (RAG), ensuring transparency and coherence. Using techniques like graph representation and models such as Sinkhorn for text ordering, and employing Natural Language Inference (NLI) for training, the research will address socio-demographic and ideological biases, benefiting from community-produced resources.
This project takes place in the context of large language models (LLMs) and conversational systems (e.g. ChatGPT, WebGPT), which have experienced tremendous practical progress in the last few months. The project GUIDANCE aims to conduct research on General Purpose Dialogue-assisted Digital Information Access, specifically how to enable users to access digital information, with the goal of overcoming several limitations of current LLMs: LLMs were not designed with Information Access whether at the level of pre-training tasks or fine-tuning ones LLMs have limited generalization abilities to new domains and/or languages; The veracity and truthfulness of the output are questionable. Potentially state-of-the-art LLMs models are not open access and the scientific methodology and proper evaluation are barely described in the scientific literature. From a community building perspective, GUIDANCE project aims at federating the Information Retrieval (IR) French Community project, by bringing together experts of the field to advance the development of Dialogue-based Information Access (DbIA) models leveraging LLMs.
In recent years, there has been an increasing demand for better sentence representations, whether for Natural Language Processing (NLP) tasks (e.g. machine translation), speech processing (e.g. speech recognition or translation) or for multimodal tasks that include Vision & Language such as image captioning or image generation from text. All the approaches rely on a representation space that should bridge the various modalities and languages. Better multilingual representations and alignments during pre-training, will ease the downstream training for NLP and speech processing tasks, by having a common representation space for all languages. Multimodal pre-training for multilingual representations could help achieve a better alignment between different languages and modalities by grounding the representtion into the visual world.
Neural models for Information Retrieval (IR) have rapidly evolved during the last few years and are now challenging the more established statistical/semantic methods. Nonetheless, neural models require large quantities of data in order to be effective and their adaptation to data (i.e. 'Zero-Shot Learning') from a different domain/language isn't direct. For Sinequa, this adaptability is key. The objective of this project is to propose methods leveraging neural models on new collection of documents, without any prior learning, and to continuously finetune these models using weak signals from users such as clicks.
Deep learning techniques have achieved remarkable progress across a wide range of domains. Despite these advances, state-of-the-art models still exhibit important limitations on certain tasks. In particular, while text and image processing have benefited greatly from model scaling and architectural innovations, reasoning over tabular data remains comparatively underexplored and less effective. This thesis addresses this gap by examining the generalization capabilities of deep learning models in Semi-Structured Table Reasoning. We show that, despite their size and sophistication, current models struggle to generalize across diverse table types and structures. A central goal of this work is to better define what generalization should mean in the context of tabular reasoning and to identify the key factors that limit it. Building on our analysis of failures, we propose three complementary research directions aimed at overcoming these limitations: 1. Representation How should tabular data be encoded to preserve structural information and support robust reasoning? 2. Generation What is the optimal trade-off between generating answers directly and leveraging external computation mechanisms? 3. Data Can we automatically generate high-quality, diverse training data to improve generalization across table-related tasks? Along these lines, we develop new methods for table encoding, reasoning via external code execution, and automatic data generation. Together, these contributions advance our understanding of generalization in table reasoning and suggest practical steps toward more reliable and transferable semi-structured reasoning systems.
In the era of big data, information retrieval (IR) plays a central role in how information is accessed and consumed. Recent advances in Transformer-based neural models have substantially improved retrieval performance. Two major paradigms have emerged in this context: learned sparse retrieval, which represents texts using weighted vocabulary terms, and generative retrieval, which formulates retrieval as the generation of document identifier. While both approaches have shown strong performance, they also exhibit important limitations. Sparse retrieval methods are often constrained by the fixed vocabulary of the underlying language model, limiting their adaptability, whereas generative retrieval methods rely on arbitrary document identifiers that tend to generalize poorly to unseen documents. In this thesis, we explore how these two paradigms can be combined to obtain more efficient and more effective retrieval representations. Our core idea is to construct sparse retrieval vocabularies from learning rather than from predefined lexical tokens. We first propose REFERENTIAL and HotBERT to investigate the use of hierarchical structured identifiers as the vocabulary representation for retrieval, whose coarse-to-fine representation is designed to capture global semantics at higher levels and progressively refine finer-grained distinctions. While this representation proves expressive and effective, our analysis reveals that directly learning and optimizing hierarchical identifiers is challenging in practice. Motivated by this observation, we introduce SAE-SPLADE, a sparse retrieval framework built on sparse autoencoders (SAE), which is an architecture that learn sparse, interpretable latent representations. By using SAE latents as the retrieval vocabulary, SAE-SPLADE removes the dependence on fixed token vocabularies and improves flexibility and representation capacity. Finally, recognizing the efficiency challenges HotBERT, we propose a theoretically lossless token-pruning method for late interaction models that reduces computation while preserving retrieval performance.
The field of Document Understanding, which addresses the problem of solving an array of Natural Language Processing tasks for visually-rich documents, faces challenges due to the complex structures and diverse formats of documents. Real-world documents rarely follow a strictly sequential structure. The visual presentation of a document, especially its layout, conveys rich semantic information, highlighting the crucial need for document understanding systems to include multimodal information. Despite notable advancements attributed to the emergence of Deep Learning, the field still grapples with various challenges in real-world applications. This thesis addresses two key challenges: 1) developing efficient and effective methods to encode the multimodal nature of documents, and 2) formulating strategies for efficient and effective processing of long and complex documents, considering their visual appearance. Our strategy to address the first research question involves designing approaches that rely only on layout to build meaningful representations. Multimodal pre-trained models for Document Understanding often neglect efficiency and fail to fully capitalize on the strong correlation between text and layout. We address these issues by introducing an attention mechanism based exclusively on layout information, enabling performance improvement and attention sparsification. Furthermore, we introduce a strategy based solely on layout to address reading order issues. While layout inherently captures the correct reading order of documents, existing pre-training methods for Document Understanding rely solely on OCR or PDF parsing to establish the reading order of documents, potentially introducing inaccuracies that can impact the entire text processing pipeline. Therefore, we discard sequential position information and propose a model that strategically leverages layout information as an alternative means to determine the reading order of documents. In addressing the second research axis, we explore the potential of leveraging layout to enhance the performance of models for tasks related to long and complex documents. The importance of document structure in information processing, particularly in the context of long documents, underscores the need for efficient modeling of layout information. To fill a notable void in resources and approaches for multimodal long document modeling, we introduce a dataset collection for summarization of long documents with consideration for their visual appearance, and present novel baselines that can handle long documents with awareness of their layout.
Information Retrieval is undergoing a paradigm shift. Keyword-based approaches, which have withstood the test of time, are challenged by a new generation of neural retrievers based on large Pre-trained Language Models. Such approaches can faithfully represent the content of documents and queries beyond the words they use -- delivering the promises of a truly semantic search experience. As models grow larger, they also become more opaque. This is a true obstacle to their widespread adoption in commercial search engines, which are increasingly faced with providing users with transparent, trustworthy, and explainable results. In this thesis, we first propose an original approach to the ad-hoc retrieval problem by learning how to represent queries and documents as sparse vectors in the vocabulary space. This results in a model that is effective, efficient, robust, and whose representations can be interpreted by design. We then propose to analyze neural ranking models from an Information Retrieval perspective, by focusing on lexical match and term importance. We first show how ColBERT - a state-of-the-art approach - relies on such aspects despite its semantic nature. We additionally extend the findings to other models by showing how the ability to perform keyword matching is architecture-dependent and heavily influenced by the presence of query terms in the training set - questioning the generalization capabilities of neural ranking models when it comes to exactly matching important query terms.
While today’s search engines work well for simple queries, there are situations where search results are not satisfactory. To cope with such situations, user-machine interactions have increased significantly since the early days of retrieval systems. The exchanges between users and retrieval systems during search sessions may contain information that is critical to the success of the information search. Meanwhile, the Transformer-based architectures relying on the attention mechanism have led to great improvements in several NLP tasks, such as summarization or translation. The architecture was soon applied to other domains, including information retrieval. Several retrieval models have benefited from this architecture’s ability to focus on document and query terms to estimate their relationships. However, the majority of these works have focused on ad hoc retrieval. The goal of this thesis is to study user modeling and user-machine interactions with transformer-based models. The contributions of this thesis can be divided into two parts, those related to user modeling and those related to interactive systems. In the former, I analyze existing user models, and in particular the text generation process of transformer-based models for query suggestions (Mustar, Lamprier, and Piwowarski, 2020; Mustar, Lamprier, and Piwowarski, 2021). In the latter, I present a new user/machine interaction framework based on the studied user models (Mustar, Lamprier, and Piwowarski, 2022).
Capturing concepts' interrelations is a fundamental of natural language understanding. It constitutes a bridge between two historically separate approaches of artificial intelligence: the use of symbolic and distributed representations. However, tackling this problem without human supervision poses several issues, and unsupervised models have difficulties echoing the expressive breakthroughs of supervised ones. This thesis addresses two supervision gaps we identified: the problem of regularization of sentence-level discriminative models and the problem of leveraging relational information from dataset-level structures. The first gap arises following the increased use of discriminative approaches, such as deep neural network classifiers, in the supervised setting. These models tend to collapse without supervision. To overcome this limitation, we introduce two relation distribution losses to constrain the relation classifier into a trainable state. The second gap arises from the development of dataset-level (aggregate) approaches. We show that unsupervised models can leverage a large amount of additional information from the structure of the dataset, even more so than supervised models. We close this gap by adapting existing unsupervised methods to capture topological information using graph convolutional networks. Furthermore, we show that we can exploit the mutual information between topological (dataset-level) and linguistic (sentence-level) information to design a new training paradigm for unsupervised relation extraction.
For language generation tasks, training regimes based on Maximum Likelihood Estimation (MLE) suffer from known limitations, often leading to poorly generated text sequences. At the root of these limitations is the so-called exposure bias, which corresponds to a mismatch between training and inference conditions. This is exacerbated by a too strong focus on ground truth reference texts, especially for tasks with a very large number of possible correct outputs, e.g. Summarization. In this thesis, we discuss how Reinforcement Learning (RL) can help to overcome these issues. RL enables us to consider complex, possibly non-differentiable, metrics that can globally assess the quality and relevance of the generated outputs. As a first illustration, we present a new metric based on Question Generation and Answering that correlates significantly better with human judgements than standard metrics. We show how the optimization of this metric can improve the quality of the generated text. Then, we move on to dynamic approaches where the metric is a learned discriminator, hence avoiding the definition of expert metrics. Previous attempts to apply Generative Adversarial Networks (GAN) for language generation tasks obtained limited success. Due to the discrete nature of text, the approaches proposed so far, based on Reinforcement Learning, have been shown to suffer from important instability issues. We discuss how alternate exploration strategies for language GANs, inducing smoother learning dynamics, can help to circumvent these limitations. We also present new algorithms to improve the decoding at inference time beyond standard decoding schemes (e.g. BeamSearch). We show how to empower beam search by considering a discriminator that cooperates with the generator. Then, we present a more sophisticated Cooperative Decoding method based on Monte-Carlo Tree Search (MCTS). Finally, we propose SelfGAN, a new GAN framework for discrete tasks, where the discriminator can pass its signal to the generator, without RL and its intrinsic instabilities. SelfGAN leverages the Cooperative Decoding methods to pass the discriminator signal in a novel way that doesn't require a reward, allowing to mitigate the inherent instability present in standard RL.
In the last decade, the evolution of Deep Learning techniques to learn meaningful data representations for text and images, combined with an important increase of multimodal data, mainly from social network and e-commerce websites, has triggered a growing interest in the research community about the joint understanding of language and vision. The challenge at the heart of Multimodal Machine Learning is the intrinsic difference in semantics between language and vision: while vision faithfully represents reality and conveys low-level semantics, language is a human construction carrying high-level reasoning. One the one hand, language can enhance the performance of vision models. The underlying hypothesis is that textual representations contain visual information. We apply this principle to two Zero-Shot Learning tasks. In the first contribution on ZSL, we extend a common assumption in ZSL, which states that textual representations encode information about the visual appearance of objects, by showing that they also encode information about their visual surroundings and their real-world frequence. In a second contribution, we consider the transductive setting in ZSL. We propose a solution to the limitations of current transductive approaches, that assume that the visual space is well-clustered, which does not hold true when the number of unknown classes is high. On the other hand, vision can expand the capacities of language models. We demonstrate it by tackling Visual Question Generation (VQG), which extends the standard Question Generation task by using an image as complementary input, by using visual representations derived from Computer Vision.
Research looking at the interaction between language and vision, despite a growing interest, is relatively underexplored. Beyond trivial differences between texts and images, these two modalities have non overlapping semantics. On the one hand, language can express high-level semantics about the world, but it is biased in the sense that a large portion of its content is implicit (common-sense or implicit knowledge). On the other hand, images are aggregates of lower-level information, but they can depict a more direct view of real-world statistics and can be used to ground the meaning of objects. In this thesis, we exploit connections and leverage complementarity between language and vision. First, natural language understanding capacities can be augmented with the help of the visual modality, as language is known to be grounded in the visual world. In particular, representing language semantics is a long-standing problem for the natural language processing community, and to further improve traditional approaches towards that goal, leveraging visual information is crucial. We show that semantic linguistic representations can be enriched by visual information, and we especially focus on visual contexts and spatial organization of scenes. We present two models to learn grounded word or sentence semantic representations respectively, with the help of images. Conversely, integrating language with vision brings the possibility of expanding the horizons and tasks of the vision community. Assuming that language contains visual information about objects, and that this can be captured within linguistic semantic representation, we focus on the zero-shot object recognition task, which consists in recognizing objects that have never been seen thanks to linguistic knowledge acquired about the objects beforehand. In particular, we argue that linguistic representations not only contain visual information about the visual appearance of objects but also about their typical visual surroundings and visual occurrence frequencies. We thus present a model for zero-shot recognition that leverages the visual context of an object, and its visual occurrence likelihood, in addition to the region of interest as done in traditional approaches. Finally, we present prospective research directions to further exploit connections between language and images and to better understand the semantic gap between the two modalities.
The increasing use of social and sensor networks generates a large quantity of data that can be represented as complex graphs. There are many tasks from information analysis, to prediction and retrieval one can imagine on those data where relation between graph nodes should be informative. In this thesis, we proposed different models for three different tasks: - Graph node classification - Relational time series forecasting - Collaborative filtering. All the proposed models use the representation learning framework in its deterministic or Gaussian variant. First, we proposed two algorithms for the heterogeneous graph labeling task, one using deterministic representations and the other one Gaussian representations. Contrary to other state of the art models, our solution is able to learn edge weights when learning simultaneously the representations and the classifiers. Second, we proposed an algorithm for relational time series forecasting where the observations are not only correlated inside each series, but also across the different series. We use Gaussian representations in this contribution. This was an opportunity to see in which way using Gaussian representations instead of deterministic ones was profitable. At last, we apply the Gaussian representation learning approach to the collaborative filtering task. This is a preliminary work to see if the properties of Gaussian representations found on the two previous tasks were also verified for the ranking one. The goal of this work was to then generalize the approach to more relational data and not only bipartite graphs between users and items.