Automatic sign language processing: the human sciences join forces with AI

“Working on the automatic processing of sign language requires advanced skills in linguistics and machine learning, but also a real ability to consult the deaf community.” In one sentence, Sam Bigeard sums up the methodological and ethical issues at stake in his research. As a linguist by training, he leads the COLaF – Défi Inria project, run jointly with the ALMAnaCH project team at the Inria Paris Centre, which aims to contribute to the development of free corpora and tools for the languages of France, taking into account their textual, oral and signed forms.

“Sign language (SL) is not a simple transposition of spoken language, simply matching a gesture to a word. It has its own grammar and specific history.” It is based on a combination of visual elements: the position and movement of the speaker’s hands and facial expressions. This multimodal character makes machine translation processes more complex than they already are for spoken languages, and even more so when switching from one SL to another. “Sign languages were not created to correspond to an official language; many emerged when a deaf community was being formed. As a result, they are not particularly standardised.”

From sign language to lesser-known languages

In this context, Sam Bigeard’s research focuses on the multilingual Wordnet for sign languages, a dictionary that brings together more than 11,000 signs in eight European sign languages (**), aligned with the synset identifiers (sets of synonyms that represent a specific meaning of a word) from the Open Multilingual Wordnet. “The aim is not only to associate signs with their French translations, but also to identify the different meanings that the word in question can have in order to establish connections between each of them and the corresponding sign in a particular sign language.”

The tool is accessible to the general public and is of particular interest to researchers working on natural language processing (NLP), such as Guilhem Fauré, whose thesis on the translation of spoken language into sign language is co-supervised by Sam Bigeard. The aim is to train models inspired by those designed for machine translation between spoken languages by exploring recent deep learning methods, such as contrastive learning, a technique that teaches the model to distinguish between similar and dissimilar examples. “Visually, the model is represented in the form of body articulations, whose time series must be predicted based on the sentence to be translated”, explains the PhD student. While the first step aims to obtain an accurate word-sign translation, the next step will focus on improving the expressiveness of the virtual signer by fine-tuning its vector representation. The ultimate goal is to establish a process that is not limited to the SL studied, but could be extended to all others, and even used to study other lesser-known languages.

Models that are difficult to train

There is one major obstacle to this endeavour: a lack of data. While training machine learning models requires very large databases, “the best corpus in French sign language is around 80 hours long, and the best in German is around 50 hours long, whereas those for spoken languages number in the hundreds of thousands of hours.” This low amount of data is primarily due to the commercial challenges associated with the development of AI. “NLP models focus on the five or six most widely spoken languages out of the 7,000 counted worldwide”, says Sam Bigeard.

Although some models claim to take hundreds of languages into account, their results for the least common languages are often mediocre. “Part of the problem is that these tools are developed by engineers who are working in languages that they themselves do not speak.” Beyond that, there is the question of validating the developed model. “In the case of text-to-text translation, it is possible to evaluate the number of incorrect characters or the ratio of correct synonyms. It is much more complicated when it comes to validating the accuracy of a sign whose meaning varies depending on the height of the hand,” says Sam Bigeard, who believes that “these metric difficulties are far from being resolved.”

Involving the deaf community

In theory, the effectiveness of the learning process could be measured by comparing the predicted coordinates of the avatar’s joints with those of the reference model. However, this evaluation system is not totally reliable, as Guilhem Fauré points out: “If the amplitude of a movement is greater, the model will logically consider it to be non-compliant from a metric point of view, even though it may be perfectly correct from a semantic point of view. And vice versa.” Involving deaf people or those who are fluent in sign language could be a way of solving this problem. 

Although he is not bilingual, Sam Bigeard says he signs “enough to be able to make an initial assessment of the videos”, but adds that it is “essential to involve deaf people throughout the development process and not just during the final validation”. Connections have been formed with the Institut national des jeunes sourds de Metz and the European Union of the Deaf. This involvement is all the more important as the work is intended to be developed into practical applications over the long term.

An alliance between the human and social sciences and AI

While the success of this endeavour hinges primarily on the expertise of the two researchers, it also requires them to share their expertise, in line with the multidisciplinary principles promoted by Inria. However, collaboration between social sciences and computer science remains rare. Firstly, because specialisation in research fields leads to a silo effect. Secondly, because differences in methodologies or ethical approaches do not facilitate spontaneous collaborations, which Sam Bigeard describes as “very enriching”.

“Despite their importance, fundamental questions about the motivations behind AI-related research and its societal impact are not sufficiently addressed in computer science research”, he notes. For his part, Guilhem Fauré acknowledges that the multiple points of view “not only make it possible to identify the most promising avenues of research, but also to draw attention to factors that are difficult to discern when focusing primarily on the code”. The strength of their association lies in the way each member contributes to overcoming differences in vocabulary in order to achieve mutual understanding. This again is a question of language and translation.

(*) The project team Multispeech is a joint venture between the CNRS, Inria and the University of Lorraine based at the University of Lorraine Inria Centre and the Lorraine Laboratory for Research in Computer Science and its Applications (CNRS/University of Lorraine)

(**) French, British, Swiss German, German, Dutch, Polish, Greek and Swedish.