Image processing and analysis

New Paradigm for Compressing Visual Data

Date:
Changed on 28/11/2022
With the number of videos uploaded to the web fast soaring, the storage of this cornucopia of visual data is now having a huge energy footprint. It thus becomes an environmental issue unto itself. Available compression algorithms cannot alleviate a pressure of such magnitude for they all aim at faithfully reproducing the input data. Hence the need for a resolutely out-of- the-box paradigm delivering much higher ratios. One way to go would consist in only encoding a data collection to a compact description of semantic nature and then reinventing ―so to speak― part of the data at the decoding phase. The feasibililty of such generative compression is currently being studied at Inria, the French institute for computer science.
Photographie d'un internaute qui regarde une série Netflix
@unsplash

Overconsumption of videos

Portrait photo Thomas Maugey
  Thomas Maugey is Sirocco member, Inria, CNRS and University of Rennes team project, in Irisa.

Every single minute, roughly 500 hours of video are uploaded to YouTube alone. In the same time span, about 700,000 hours are being streamed. Not to mention the other sharing platforms and social networks of all ilk. This mind-boggling tsunami of data translates into tremendous amounts of electricity being gobbled to power and cool down the storage facilities.

To cap it all, the trend is just skyrocketing. Reducing the volume of data through compression would contribute to mitigate the problem. But classical algorithms can only offer limited gains due to an inherent limitation. «They are meant to reproduce the input data as faithfully as possible. Therefore, if we are to shrink the size of the data drastically, we need to shift paradigm. We need to change the very goal of compression itself» sums-up Thomas Maugey, a scientist at Inria research center, in Rennes, Brittany, France.

In the wake of an initial Inria exploratory action, Maugey was recently granted funding by the French National Research Agency in order to pursue an investigation into what has now been coined generative compression. The overall project goes by the acronym DARE which stands for Data Repurposing. 

Like a Music Sheet

«Instead of resembling as much as possible to the input data, we will allow ourselves not to reproduce it, but rather to encode it to a compact description of a semantic nature. The best analogy that comes to mind is the music sheet. When you think about it, the most compact, the most compressed representation of music is none other than... the music sheet itself. It can be replayed. Granted, the rendering might not be exactly identical to the original recording, but it is still quite close. So, the idea is to come up with the same concept of music sheet but for images. The data will then be reinvented at the decoding phase so that the regenerated content will be semantically coherent with the initial one. It will look like the input data.» 

For doing so, scientists will leverage Artificial Intelligence, and more specifically a type of neural network called Generative Adversarial Network (GAN). This algorithm learns to generate new data with the same statistics as the training set. For instance, it can create strikingly realistic photographs of people who do not even exist. And these photos are not composites of several people whose chin, jaws and eyebrows would be somehow stitched together. They are pure random creation. But in order for GANs to do that, they need something as an input. «They are fed with, say, 1,000 random digits. And from these digits, they will invent a portrait.»

However, there might be another way to use them. «If we could find a link between these digits and the rendered output, then we could pilot this generative instrument. Finding this link, building the input description, tuning the semantic is what our research is all about.» 

More Suitable for Cold Data

At the end of the day, the compression ratios could be gigantic. Having said that, «generative compression would certainly not be suitable for all videos. One wouldn't want to use it for the material of an artistic nature or recordings loaded with affect. Family footage for instance. It could be more relevant for cold data: all those videos that are very seldom accessed, that you don't really care much about anymore, but that you haven't resolved yourself to delete yet.» Repurposing thus offers a third way in between keep and trash. 

Additionally, the new compression scheme could also be applied to only one part of the image. «You wouldn't touch the foreground where the action is taking place but regenerate the background where nothing happens and to which nobody pays attention anyway. Keep the soccer players on the field, but regenerate the crowd in the stadium, for instance.» 

Who will ultimately decide which compression scheme is to be applied to which data? «It's up to users to make that choice. Our data repurposing framework will include tools for helping them to compress elements of their video collection as they see fit, be it for social networking, storage on the cloud, so on and so forth.»

Titre

To raise awareness about the environmental impact of people's digital activity.

Verbatim

We will have a website through which everyone will be able to assess their own footprint. For instance, what's the energy cost of downloading a series from Netflix or uploading videos to the cloud. It will pinpoint the energy consumption step by step all along the chain and simulate various compression schemes.

 

Auteur

Thomas Maugey

Poste

Scientific leader

The attitude of the public turns out to be an important ingredient in this quest for frugality. «There is a risk of a rebound effect, Maugey points out. Oftentimes, when technology improves the efficiency with which a resource is employed, the total consumption of that resource increases rather than decreases. As engine gas consumption went down, the usage of car boosted. The same thing is occurring with videos. The easier they are to make, the more they end up online.»

Titre

Le projet Madare en podcast (in French)

visuel audio bloc par defaut
Titre du lecteur

Podcast de Thomas Maugey

Fichier audio
Audio file