Processing Real-World Light Field Images
Epitomized by the Lytro, a new breed of so-called plenoptic or light field cameras allow to simultaneously capture multiple images of one single scene. This rich visual material can be edited in order to pick from different possible focuses, different depths of field and even different lens apertures. However, the current software line-up would be hard-pressed to process such files due to their dimensionality and complexity. Inria scientist Christine Guillemot has been awarded an Advanced Grant by the European Research Council (ERC) in order to tackle this unprecedented challenge and build algorithms for next-generation photo and video applications using light fields.
Eversince the early days of photography, back in 1839, capturing a scene has been all about getting light rays to converge through a lens on a sensible surface, be it the chemical emulsion of yore or nowadays digital sensor. The recorded image is thus the sum of all the light rays emitted by one point over the lens aperture. In the last 5 years, reviving the old concept of ‘integral photography’ first introduced by Nobel Prize Gabriel Lippmann back in 1908, two startups, Raytix and Lytro, have made their debut on the market, unveiling the first real plenoptic cameras to speak of.
In contrast to conventional cameras and their ‘ordinary’ lens, these innovative devices feature a digital sensor coupled with a micro-lens array containing thousands of miniature lenses. They have come to be known as light field cameras for they capture not only the intensity and the position of the light, but also the geometric distribution of the rays passing through the lens, which is lost in traditional photography. Although these pioneer products are not flying off the shelves yet, they hold great promises and their gamechanging potential wasn't missed by the industry. Nikon, Canon, Sony and Apple stand among the prominent companies that have been recently granted patents for such technology.
“The light field —in orther words the recorded flow of rays— yields a very rich description of the scene, enabling advanced creation of novel images from a single capture,
says Christine Guillemot, head of Sirocco research team at Inria center in Rennes, Brittany, France. It allows simulating a capture with a different focus and a different depth of field,
” which can help to save out-of-focus shots. “It also permits simulating lenses with large aperture,”
which is a huge plus in low-light conditions or for creating the artistic background blur known as ‘bokeh’. Similarly of interest is the possibility of producing 3D views. “The light field can be seen as capturing multiple viewpoints of the scene, giving information about the parallax and depth of the scene.
” Considering such amazing capabilities, light field cameras in the future “are likely to bring disruptive changes to the field of computational imaging with tremendous impact in a number of application domains.
Having said that, the path to full deployment is strewn with several daunting hurdles. One of them is the paltry resolution of current plenoptic sensors. For instance, the recent Lytro Illum only yields 2 megapixels images. Another barrier is the huge amount of captured high dimensional data: 4D for still image and 5D for video. It has “obvious implications on the end-to-end processing chain: compression, communication, rendering. ” In addition to that, “a third barrier relates to the possibility for the users to edit and manipulate light fields as they can today with 2D images and videos. All these obstacles translate into challenging processing problems which need to be addressed before being able to fully exploit the potential of this technology. Such barriers cannot be effectively overcome by simply applying the models and methods which have made the success of digital imaging in past decades. They call for new algorithmic foundations ”
And that is precisely the purpose of CLIM (Computational Light Field IMaging), the 5-year research project for which the scientist is being granted 2.5 M€ by ERC. It will address three specific challenges. “From the very high redundancy and correlation which light fields data exhibit, it is apparent that the data have an intrinsically low-dimensional structure although they live in a high-dimensional ambient space. So the first goal will be to develop methods to discover and characterize these lower-dimensional spaces and find sparse representations.
Second challenge: “designing a storage and transmission bandwith efficient coding system which would fully exploit and preserve the geometrical models and structures of the light fields data. This is the core of our research. ” Last, but not least, the third aspect regards the “development of algorithms for compressed light fields computational imaging in order to enable advance features such as refocusing, perspective shifts, extended focus with high resolution, panoramic imaging and editing. ”
Anticipating “a strong impact of this research on the digital sector, ” Guillemot believes it will “open new horizons for computational imaging applications in a variety of sectors. ” And five years from now, part of this work will hopefully contribute to new image standards such as JPEG-Pleno.