Ginger Delmas
Looking for an AI Research Scientist position.
I recently obtained my PhD, from IRI-UPC-CSIC & NAVER LABS Europe, where I was supervised by Francesc Moreno-Noguer, Philippe Weinzaepfel and Grégory Rogez.
I gathered some expertise in linking Natural Language with other modalities (images, 3D human poses) for multi-modal retrieval and generation, with particular application in human pose modeling tasks.
Papers
PoseEmbroider: Towards a 3D, Visual, Semantic-aware Human Pose Representation
The PoseEmbroider framework combines multi-modal views of the human poses (images, textual descriptions, 3D joint rotations) to derive a rich visual-, semantic-, 3D-aware representation, which can be reused off-the-shelf in various downstream tasks (eg. any-to-any multi-modal retrieval, pose estimation, pose instruction generation). Its transformer core is trained in a contrastive fashion, and is shown to outperform the standard multi-modal alignment baseline.
PoseFix: Correcting 3D Human Poses with Natural Language
We introduce the PoseFix dataset, which consists in over 6k triplets of 3D human pose pairs and a text modifier describing how the source pose needs to be modified to obtain the target pose. We further train a text-based pose editing model to generate corrected 3D body poses given a query pose and a text modifier; and a correctional text generation model, where correctional instructions are generated based on the differences between two body poses.
PoseScript: 3D Human Poses from Natural Language
We collect a dataset, PoseScript, pairing 3D human poses from AMASS and descriptions both written by human annotators and generated automatically by our proposed pipeline. We use PoseScript to train text-to-pose models, both for retrieval and generation. Pretraining on automatic data boost performance by a factor 2.
ARTEMIS: Attention-based Retrieval with Text-Explicit Matching and Implicit Similarity
We take inspiration from image retrieval and cross-modal retrieval to tackle the task of composed image retrieval: we design two complementary modules, each focusing on one modality of the query. The Explicit Matching module assesses how potential targets fit the textual modifier while the Implicit Similarity module compares potential target images to the reference image, assisted by the text. We validate our approach on FashionIQ, Shoes and CIRR.
Thesis
Linking Human Poses With Natural Language
Human pose is key to multiple human-centric applications (art, sport, embodied AI...). Until recently, researchers had addressed underlying tasks where human pose is mostly studied in conjunction with images. The arrival of efficient language models fostered the incorporation of linguistic in vision frameworks, and thereby powered multi-modal applications. This thesis fits into this dynamic. We aim to leverage Natural Language (NL) to bud human pose understanding in human-centric tasks. In contrast to prior endeavors, we juggle with static 3D human poses, images and detailed NL texts all together. We further explore novel multi-modal applications, requiring fine-grained understanding of the human pose.
First, to alleviate the lack of data, we introduce new datasets linking 3D human poses with NL texts. We notably investigate two settings. One where the text is a description of the target pose, and another where the text provides modification instructions to reach the target pose from a source pose. These datasets result both from (i) the collection of crowd-sourced annotations, and (ii) the automatic, rule-based generation of texts, which consists in the incorporation of classified pose measurements into templates sentences.
Next, we use these datasets to develop several cross-modal generation models like text-driven pose synthesis, pose captioning, text-guided pose editing and generation of textual posture feedback. Eventually, we connect 3D, text and images through a novel combinating framework, so as to derive a versatile, multi-modal pose representation, to be leveraged for downstream tasks akin to pose estimation or NL posture feedback from visual input.
In summary, we tackle multiple machine learning tasks entailing human pose understanding, thanks to the connection of human pose and Natural Language.
Talks
Biography