What if audio programming became child’s play?

Embedded audio systems are among those everyday objects that we constantly use without even thinking about it. Some examples? The voice assistant in our smartphone, the talking doll or toy, Bluetooth headphones, the background music of mobile applications, or synthesisers, among others. “The amount of audio systems is growing all the time, especially since the processors on which we install them are also expanding and offering new possibilities”, says Tanguy Risset,  a researcher at the Citi laboratory in Lyon (INSA Lyon and Inria). “For example, there are micro-controllers which cost just a few euros, or FPGA (Field Programmable Gate Arrays) which provide access to real-time audio with less than a millisecond between an instruction and sound production, a time lag which is imperceptible to the human ear.” 

There is a drawback to these processors, however; they are difficult, even very difficult, to programme for FPGA. In fact, the majority of embedded audio systems are not developed by IT professionals, but by keen amateurs, such as geeks, composers, musicians or sound engineers, i.e., “makers” who draw on new technology to produce devices for their personal needs. As for the companies who sell embedded audio systems, they overcome the complexity of programming by agreeing to devote a great deal of time to the matter.

The Lyon-based researchers at Citi and Grame (French National Centre for Musical Creation) have been working together for two years to facilitate this task for such users. Citi specialises in embedded systems, signal processing and compilation. Grame designed the Faust programming language, which is used worldwide to develop audio systems without having to manage the purely IT aspect.

“Faust, which has existed for 15 years, wasn’t originally targeted at embedded audio”, Yann Orlarey from Grame points out. We’re moving towards that direction, with compilers that translate a programme from Faust, which is easy to create, into a programme that can be executed on an embedded system. The aim is to multiply these compilers in order to cover all the processors on the market, right up to FPGA.” 

On this last point, which is by far the most ambitious, Grame, Citi and the LMFA (Fluid Mechanics and Acoustics Laboratory, affiliated with the École Centrale de Lyon, CNRS, Claude Bernard–Lyon 1 University and INSA Lyon) have received public funding since March 2021 through the French National Research Agency FAST project. They have already developed a prototype for the Faust - FPGA compiler, which they are improving step by step.

The aim, however, is to reach the general public in its broadest sense. Another project led by Grame, Amstramgrame, testifies to this. The project focuses on a programmable musical instrument for primary and secondary school students “We called this instrument the Gramophone”, Romain Michon from Inria tells us. “It’s the size and shape of a small tin. Pupils learn to programme the instrument on Faust, by using what they learn in maths and physics. Each pupil has their own Gramophone and a class can form an orchestra, or work with a musician to produce and perform original compositions.” 

In order to use these very practical applications, the team in Lyon goes through increasingly upstream research steps. “We develop, for instance, signal processing algorithms for audio”, Yann Orlarey explains. “We also explore new arithmetic techniques to ensure the accuracy of calculations without imposing manual operations, as is the case at present. All this helps to facilitate access to embedded systems.” 

The systems of the future will be surprisingly innovative. In the USA, a start-up at Stanford University is working on a version of the electroencephalogram (EEG) with sound, a sort of “brain stethoscope”. It should enable neurologists to identify brain activity much more easily than with signals traced on paper. “We discovered this project when the start-up produced a prototype of its system in Faust”,Romain Michon tells us. “Doctors detect significant EEG events by ear, events which they might not spot on a graph. Our brain is more efficient at analysing sound than image.” 

The Lyon-based team expect these studies to lead to other advances, in particular with regard to the famous FPGA. The lag of less than a millisecond between instruction and sound production, known to researchers as “ultra-low latency”, opens vast opportunities.

“Sound signal processing is becoming faster than its transmission in space”, Tanguy Risset explains. “We can initiate active sound control inside a car, for example, to cut out an unpleasant sound and generate a replacement sound. We can also create ultra-realistic sound environments for video games. We can associate sound processing to haptic feedback to give digital instruments such as guitars or drums the same feel as their acoustic counterpart.” And let’s not forget that all of this is possible with a compact, lightweight, embedded system, instead of a computer!

The computing power of FPGA also broadens the scope of WFS (Wave Field Synthesis), a technology already in use in museums; equipped with an audio guide, visitors hear the description of a work when they approach it. “WFS serves to spatialise sound, to recreate sound fields in limited spaces”,Romain Michon explains. “With a single FPGA card, we could produce high-density WFS, in other words, roll out up to a hundred small loudspeakers, to reproduce the sound as we hear it.” 

For example, the cello vibrates and emits in all directions, and this concurrent emission of multiple sounds defines a highly specific “signature”.On the other hand, if the cello is hooked up to a loudspeaker, the latter only generates one wavefront.Several small WFS loudspeakers would recreate the richness and complexity of the instrument’s sound.

One final example testifies to the potential of the Lyon-based project. We can imagine audio guides of the future which will be capable of going back in time, to immerse the visitor in the sound atmosphere which existed in a monument or a prehistoric cave, centuries or thousands of years earlier. “We already have the algorithms, we need to define the appropriate device”, Yann Orlarey underlines. “And who knows? Maybe all we’ll need is a smartphone to experience these virtual acoustics.” 

• Amstramgrame - Official launch, GRAME, French National Centre for Musical Creation, 26/11/2020  
• The “brain stethoscope” - 3-minute presentationStanford, 21/03/2018