Deep learning for new imaging methods

Date:
Changed on 01/02/2021
To support artificial intelligence research in France, the government recently provided funding for 40 research and teaching chairs. Christine Guillemot, Director of Research at the Inria Rennes – Bretagne Atlantique Centre is one of those who was awarded funding. Her work aims to allow neural networks to efficiently process the high-dimensional images generated by new plenoptic sensors and omnidirectional cameras.
représentation d'une ville en photo 360°
Image par Evgeni Tcherkasski de Pixabay

 

A single click to smooth out a few wrinkles, another to slim the figure, a third to add a little colour… The job’s done! In just a few years, artificial intelligence (AI) has become an integral part of image processing with astonishing results. But the famous neural network algorithms behind this success are confronted with a problem: they struggle to integrate the stunning dimension of the new 3D images produced by omnidirectional cameras or plenoptic sensors. These so-called "light field" sensors no longer simply transcribe the image in its 2D plane. They incorporate every dimension of the scene, including volume, allowing you to choose one focus point among several thousands, modify the depth of field or even change the framing.

Christine Guillemot is the head of Sirocco, a team specialising in signal processing and representation, particularly in the field of image processing The AI Chair is called DeepCIM, an acronym for: Deep learning for Computational Imaging with emerging image Modalities.

The artificial intelligence chair held by Christine Guillemot  aims to develop neural networks able to work on this scale. It follows on from a project CLIM (Computational Light Field IMaging) on these new imaging methods chosen by the European Research Council (ERC) in 2016.

The size of the data impacts the algorithms

“We had already started to work on deep learning during the ERC project. These methods work well but we can’t immediately transpose them to omnidirectional or plenoptic imaging. Firstly, because the size of the data also impacts the size of the algorithms used for learning. The neural networks already have to incorporate millions of parameters in 2D. This can cause problems, for example, for storage on mobile devices. Now, with the new imaging methods, the number of criteria to be taken into account explodes. These new images also have specific properties that must be taken into account. For example, the omnidirectional image is spherical. We must therefore also integrate this into our models.

appareil plénoptique Lytro

 

The new research will first consist in articulating learning and dimensionality reduction. “We suppose that the data can be represented by smaller data in subspaces, which are specific domains usually defined by dictionaries or that will be learned by neural networks.”  Here, the research will use a whole range of methods that are traditionally used in signal processing and compression such as parsimonious modelling, low-rank approximation and graph-based models. “Plenoptic data can be seen, for example, as multi-view captures. The graphs will help us connect the correlated points. Thanks to the depth information of the scene, we know that a pixel with an xy position in one view is correlated with an x'y' pixel in another. But the two pixels actually correspond to the same 3D point in space. This dependency information can be represented by a line connecting two points of a graph to allow the neural network to then take it into account.

Titre

Inverse problems

Verbatim

An inverse problem is a poorly posed problem because there is no single solution given the input observations.

The topic is also closely linked to the resolution of inverse problems as they are classically encountered in signal reconstruction: denoising, deblurring, super-resolution, compressed sampling etc. “An inverse problem is a poorly posed problem because there is no single solution given the input observations. These can be blurred or disrupted by noise if the sensor is of poor quality, if there is optical aberration or movement during capture. Depending on the device used, different observations can therefore be made from which the imaging model must trace back to the data of the real scene.

For the time being, "in the literature, we learn a neural network for each type of inverse problem. One for denoising, one for super-resolution, etc. We would like to obtain a single network that can be used for all these cases. It would be a question of representing an a priori on the image, on the data we are aiming to reconstruct, and being able to use this deep a priori in traditional signal processing algorithms to combine the optimisation methods.”

There is an additional difficulty, however: “to function well, neural networks must be supplied with large amounts of learning data. Millions of examples. But plenoptic and omnidirectional technologies are recent. We don’t have that much learning data yet. We therefore risk having parameters that stick too closely to a small number of images. This is known as over-interpretation or overfitting.

 

Plenoptic microscopes

The Chair will last four years. In addition to Christine Guillemot’s work, it will finance, via the French National Research Agency, a young researcher, a post-doctoral researcher, a PhD student and an engineer. There will also be a PhD student funded by Inria and two students carrying out Cifre PhDs in two partner companies operating in the image sector.

The Cifre system offers companies financial support in order to recruit a PhD student whose research work will lead to the defence of a thesis. Expenses may be eligible for the research tax credit (CIR) under certain conditions.

This work concerns many fields of application, including driverless cars. “Vehicle navigation is based on computer vision. The neural network is used to interpret the content in the scene. With plenoptic images, we can collect much more in-depth information than in 2D.” 

In a completely different vein, "in microscopy, the 3D information will allow us to use super resolution to choose the best focal plane in the optical axis. At Stanford University, plenoptic microscopes are already at the prototype stage.

 

Christine Guillemot's portrait

Portait de Christine Guillemot

The rank of Officer of the Legion of Honour

In 2010, Christine Guillemot was awarded the distinction of Knight of the Legion of Honour. On 1 January 2021, she was promoted to the rank of Officer. This order of merit recognises the very fruitful career of the internationally recognised image specialist.

Click here to learn more

2 hommes et une femme de dos travaillent sur un projet dans une cafetariat

Head of the Sirocco research team

Christine Guillemot is in charge of Sirocco research team at Inria Rennes - Bretagne Atlantique and at IRISA.

She is interested in the design of algorithms in the fields of analysis, modelling, representation, compression and communication of visual data (images, 2D and 3D video), for applications in areas such as 3D TV, multi-camera sensor networks and satellite imagery.

vieux poste de télévision encadré par des vinyles et des cassettes VHS

Career before Inria

She obtained her PhD at the ENST (Ecole Nationale Supérieure des Télécommunications) in Paris. From 1985 to 1997, she worked at France Telecom / CNET, where she was involved in various projects in the field of compression for TV, HDTV and multimedia applications.

Between 1990 and 1991, Christine Guillemot also worked at Bellcore (USA) as a guest researcher.

facade du centre Inria Rennes

Arrival at Inria

Inria Research Director since 1997, she has been head of the TEMICS project-team at the Inria centre in Rennes from 1999 to 2011, then of SIROCCO created in January 2012. In 2015, she was awarded an ERC Advanced grant for the CLIM (Computational light field imaging) project, for a period of five years. In 2020, she was awarded a research and teaching chair on artificial intelligence for the DeepCIM project (Deep learning for computational imaging with emerging image modalities).

Photo d'un prix spécial avec un chapeau de diplomé

An internationally recognised researcher

Christine Guillemot is a recognised specialist in her field of research; she has received numerous awards throughout her career :

  • 2013: IEEE (Institute of Electrical and Electronics Engineers) Fellow
  • 2015: Winner of the Google Faculty Research Award; and winner of an ERC Advanced Grant for the CLIM (Computational Light Field IMaging) research project
  • 2019: EURASIP Technical Achievement Society Award winner.