Jan Sedmidubsky and other researchers receive "Best Short Paper Award Honorable Mention"
For the scientific article were awarded doc. Jan Sedmidubsky, Ph.D.; Tomas Rebok, Ph.D. and other authors, received the prestigious "Best Short Paper Award Honorable Mention" at the SIGIR 2023 A* Conference. Their paper "Text-to-Motion Retrieval: Towards Joint Understanding of Human Motion Data and Natural Language" was among the top three papers out of 154 accepted and 613 submitted papers (always in the given category).
You can learn more about the article in the annotation directly from one of the authors, Assoc. Thanks to recent advances in computer vision, it is possible to extract a person's movement from a regular video in the form of sequences of simplified 3D skeletons. Although automated processing of such spatio-temporal human motion data offers great application opportunities in many fields, efficient content-based access to such data remains an unsolved problem. In this paper, we introduce a novel search concept that aims to find in an unannotated database of human movements those relevant to a textual query specified in natural language. As an example, a user could enter the query "a person kneeling on the ground slowly falls on his back", which then results in the 10 most relevant database movements. In addition to defining the search concept, we have presented an implementation based on the integration of text and motion modalities, namely the BERT/CLIP language model (for encoding the text modality) and a Transformer neural network (for encoding the motion representation modality). We also presented qualitative metrics and evaluated first experiments on the recently introduced HumanML3D and KIT Motion-Language datasets. We hope that this paper will be a first step towards searching videos according to the way human movements are performed.