PlantNet.The name says it all: this interdisciplinary project combines botany and computer science. It is the fruit of an encounter between two teams, each specialising in one of the two areas (see inset). What it is exactly? A collaborative platform for plant identification. “When PlantNet was launched 10 years ago, it was a very theoretical research project that aimed to use image recognition techniques to inventory biodiversity”, explains Alexis Joly, joint head of the project. The Inria researcher and Pierre Bonnet, his alter ego at Cirad (French agricultural research and international cooperation organisation), are the driving force behind the project, which started out using the means at hand: “We were mainly relying on scanned leaves”, the scientist explains. “To start with, we didn't even consider a mobile app because there weren’t many around at the time, only a website”.
From seed to fruit: the growth of a joint project
In 2009, in response to a call for projects by Agropolis Fondation, the Amap laboratory (Botany and modelling of plant and vegetation architecture) at Cirad in Montpellier, contacted the Imedia team at Inria. Overseen by Inria, Cirad, the IRD (French Research Institute for Development) and the Inrae (French Research Institute for Agriculture, Food and the Environment), the collaboration began, and PlantNet was born. The adventure continues today with the Amap team and the Zénith team at Inria Sophia Antipolis, which Alexis Joly joined in 2011.
Using algorithms to look for similarities
Others joined the team during the first R&D project, between 2010 and 2014. Hervé Goëau, a Cirad researcher in data science, also a member from the outset, was joined by three Inria engineers: Julien Champ and Antoine Affouard in 2012, then Jean-Christophe Lombardo in 2014. And the research progressed. “We focused on image recognition algorithms and in particular the search for similarities”, explains Alexis Joly. “This not only allows the name of the plant to be identified, but also to list the species that are the most similar. The user is therefore involved in the process because they choose the correct plant from among the ones presented. This improves the algorithm”.
A moment of doubt
“When we released the first version of the mobile application, it received a lot of media coverage, but the performance wasn’t up to scratch and lots of botanists were not at all convinced” explains Alexis Joly. “Thankfully, there was a community of people who understood that by sharing their images they would help the algorithm learn. Now botanists have fully adopted the tool and are our main users!” - Alexis Joly
It is a major innovation that PlantNet's competitors don’t have. The task was tedious at the beginning, but in 2015, deep learning made the researchers’ work easier. “We were among the first to develop plant identification at such as scale, listing 8,000 species”, says Alexis Joly.
The application is tested by the general public
2015 was also the year in which PlantNet took on a new dimension when the project received funding for four years through the Floris'Tic initiative, financed by the French government's Future Investment Programme, which now includes an education section. The mobile app, launched in 2013 for botanists, also became available to the general public. Now anyone can download the app and take a photo of a plant, view a list of potential species that might match and “vote” for the one that seems to be the right one. PlantNet can also be used for teaching and informing as well as agroecology, to identify things that aid growth or determine early on which plants are invasive, for example.
“We concentrated particularly on data validation”, says the joint head of PlantNet. “In a collaborative platform like ours, users participate in plant definition through their votes so the algorithm has to make decisions based on these choices: what value should they be given? How should they be taken into account?”
A serious game for data validation
To explore this important matter further, the team developed a serious game called ThePlantGame. The player is shown a species A, for example, and is asked if it is species A, B or C. “By analysing identification success and failure rates, the algorithm is able to draw up a sort of skills profile and attribute more or less weight to a vote depending on the profile” explains Alexis Joly.
PlantNet in 5 dates:
2011: first web version
2013: first mobile app
2015: integration of deep learning
2018: first study in ecology using PlantNet data
2020: integration of PlantNet data by the GBIF (Global Biodiversity Information Facility) and the Inria Innovation Prize - Académie des sciences - Dassault Systèmes
This educative technique also helps PlantNet progress. The application is thus becoming a well-known, reliable tool... and the number of users is exploding. “When we exceeded 100,000 downloads, we were amazed!” says Alexis Joly. “We reached the 20 million mark in 2013, and in spring this year, we reached around 500,000 users per day. The number doubles every year. We’re wondering how far it will go!” At the same time the data integrated into the application is increasing and now includes 27,909 plant species.
Making PlantNet a predictive tool
The team doesn’t intend to stop there. Since 2019, PlantNet has become an InriaSoft consortium and the research continues. “We’re examining the sampling bias of our data, for example”, explains Alexis Joly. “The data is collected by users who are mostly in urban areas... so certain species in these environments are overrepresented. Our aim is to make unbiased models of plant distribution”.
Other avenues of research include using deep learning to learn about the environmental preferences of plants and to predict where a certain plant is likely to grow. The team also aims to develop an embedded “offline” version of the application.
Finally, since the start of this year, the most reliable observations from PlantNet have been integrated into the GBIF (Global Biodiversity Information Facility) database which aims to provide free access to data on all forms of life on Earth to everyone everywhere. “We are the first to integrate data from artificial intelligence” says Alexis Joly. “It’s an important recognition of the level of performance of our application.” An achievement now accompanied by the Inria - French Académie des Sciences - Dassault Systèmes Innovation Award.
Find out more
- Hugo Santacreu, Project Manager for PlanNet at Cirad, explains how to use the mobile application. “Démo PlanNet”, BioAgri, YouTube, May 2016
- “10 years of PlantNet: the collaborative platform secures a long-term future by creating an open consortium", CIRAD, 26 February 2020