Period
2024 — 2028
48 months, ongoing
Role
Principal Investigator
Scientific coordinator
Funding
≈ 294 k€
ANR — "JCJC" grant
Host
IMT Atlantique
Lab-STICC, BRAIN team
In a Nutshell
Modern AI is trained on massive amounts of data, and scaling laws tell us that more data generally means better models. But more is not the only axis that matters: what a new example adds depends on how different it is from the ones already seen. ENDIVE takes that observation seriously and asks a simple question — when the budget is on the number of examples rather than on the amount of compute, how much can be gained by choosing examples, features, or models that are diverse rather than merely individually good?
The technical entry point of the project is random sampling procedures with diversity-encouraging properties, and in particular Determinantal Point Processes (DPPs). A DPP accounts both for the relevance of the points it samples and for their similarity, which yields a diverse form of importance sampling: it covers the underlying distribution better, and estimates its statistics with lower variance than independent uniform sampling.
A note on scope. ENDIVE was written for few-shot learning, at a time when learning from a handful of labeled examples was a research field of its own. Three years later, few-shot learning has largely become one regime of foundation model adaptation. This does not change the objectives of the project, but it did lead me to reorient it, giving considerably more space to large language models — and especially to discrete diffusion ones — still under the prism of diversity.
Why Diversity? A Picture
The figure below contrasts a DPP draw with a uniform draw of the same size, on the same point cloud. Ten points (colored) are sampled among five hundred (gray); the diamonds mark the mean of the sampled points against the mean of the full cloud.
Research Axes
The revised project explores the benefits of diversity along two complementary axes.
Diversity in the Data
Curation & data efficiency
Selecting training examples on individual quality or importance alone produces highly redundant subsets. Combining quality with diversity is known to let carefully curated subsets match — or beat — much larger ones.
- How can diversity-aware curation make the training of diffusion LLMs more data-efficient? A criterion designed for next-token prediction scores an example by what it teaches about the continuation of a prefix, whereas masked denoising supervises every position at once, across corruption levels: what makes an example informative is not the same quantity.
- How does data diversity shape the behavior of these models — their decoding trajectories, and the internal mechanisms they give rise to?
Diversity in the Representations
Features, contexts & models
Foundation models encode a great variety of concepts, many of them irrelevant to any particular downstream task. Diversity offers a criterion for deciding what to keep, what to ask, and whom to ask.
- Which features are relevant? Highly correlated features add little; diverse task-relevant ones may cover the information the model holds far more compactly.
- How can diversity of context be exploited? Prompting and retrieval produce several representations of the same input — complementary interpretations rather than one optimal context.
- Can diversity guide the selection and combination of foundation models, identifying complementary ones rather than the individually strongest?
Mechanistic interpretability has become the natural tool for the second axis: sparse autoencoders decompose representations into features whose overlap across models trained on different data diets is, in itself, a measurable notion of diversity.
Publications
NeurIPS 2026
Under review
Jonathan Lys, Vincent Gripon, Axel Marmoret, Lukas Mauch, Fabien Cardinaux, Ghouthi Boukli-Hacene, and Bastien Pasdeloup
A decoding framework that generalizes beam search to discrete diffusion language models, which refine whole sequences in parallel instead of extending a prefix. Candidate selection at each diffusion step is cast as MAP inference over a partitioned DPP, whose kernel combines the quality signal carried by the logits with a diversity signal read from the model's own hidden states — so the procedure stays model-internal, with no external verifier. The partition constraint prevents hypotheses from collapsing onto the same trajectory, and the greedy solver's overhead is more than offset by a ~40% cut in peak memory.
The most direct ENDIVE contribution to date: the project's DPP machinery becomes a decoding rule, and diversity is measured in the model's own representation space.
NeurIPS 2025
Yassine El Ouahidi, Jonathan Lys, Philipp Thölke, Nicolas Farrugia, Bastien Pasdeloup, Vincent Gripon, Karim Jerbi, and Giulia Lioi
A foundation model built to absorb the heterogeneity of EEG recordings across electrode layouts, devices, protocols and subjects. A 4D positional encoding represents jointly the 3D location of each electrode and the temporal position of each signal patch, which makes the electrode layout part of the model's input rather than something recordings must be projected onto. Pretrained by masked autoencoding on 60,000+ hours from 92 datasets and ~25,000 subjects, it reaches state-of-the-art results on ten downstream tasks, including under linear probing.
The concrete setting in which few-shot learning became foundation model adaptation, and the reason the project reoriented. Assembling 92 heterogeneous datasets also turned data curation into a research question of its own — feeding directly into the data-diversity axis.
EUSIPCO 2026
Jonathan Lys, Vincent Gripon, Bastien Pasdeloup, Axel Marmoret, Lukas Mauch, Fabien Cardinaux, and Ghouthi Boukli-Hacene
A training-free way of spending extra computation at inference time in a frozen transformer, by re-applying a selected range of blocks to the propagated hidden state. Naive looping degrades accuracy — re-executing layers pushes activations off the manifold seen during standard inference — so the contribution is a lightweight regularization interpolating looped states with their non-looped counterparts. With the loop region chosen once, and consistently found at 40–60% of relative depth, the gains are modest but consistent on the Gemma-2 models; pre-norm Llama-3-8B responds more variably.
Representation axis: it probes how much usable capability already sits latent in a frozen model's residual stream, and can be recovered without touching a single parameter.
EUSIPCO 2026
Jonathan Lys, Vincent Gripon, Bastien Pasdeloup, Axel Marmoret, Lukas Mauch, Fabien Cardinaux, and Ghouthi Boukli-Hacene
A structural tension in autoregressive transformers: the hidden state at a position is initialized from the current token and carried along by the residual connections, while the supervision target is the next one. Decoding intermediate states through the tied embedding matrix reveals three regimes — early layers decoding to the input sequence, intermediate layers to nothing interpretable, late layers to the one-token-shifted sequence — with the transition deep in the network (around layer 17 of 26 for Gemma-2-2B). A learned gate attenuating the residual branch concentrates on the final layer and outperforms fixed single-layer cuts, which are themselves very sensitive to the chosen layer.
Representation axis, from the interpretability side: before one can ask which features are worth keeping, one has to know where in the network a representation stops describing the input and starts describing the prediction.
Team & Collaborations
The project is coordinated by Bastien Pasdeloup at IMT Atlantique (Lab-STICC, BRAIN team), with the support of Vincent Gripon, Nicolas Farrugia and Alexandre Reiffers-Masson. Expertise on determinantal point processes is brought by Nicolas Tremblay (Gipsa-lab, Grenoble) and Rémi Bardenet (CRIStAL, Lille).
ENDIVE allowed me to hire the following researchers:
Jonathan Lys
PhD student • 2024 — 2027
Diversity in few-shot learning, then discrete diffusion LLMs. Author of the three publications above on decoding diversity and on pretrained transformers.
Google Scholar →
Post-doctoral researcher
18 months • from 2027
To be recruited, primarily on mechanistic interpretability and the diversity of the features and mechanisms models acquire.
Contact me →
Master 2 intern
6 months
To be recruited. Get in touch if diversity-aware curation or interpretability of diffusion LLMs sounds like your kind of problem.
Contact me →
Back to Research
