hwloc helps Frontier break the exaflop barrier

Changed on 20/07/2022
The American supercomputer Frontier, ranked since May 2022 at the top of the TOP500 most powerful machines in the world, owes its performance in part to Inria's hwloc software. For the first time, it has broken the mythical barrier of the exaflop, or 1 billion billion operations per second!
Illustration HPC

A supercomputer that occupies a 700 m² room

Installed in the Oak Ridge (Tennessee) laboratory of the U.S. Department of Energy, Frontier occupies a 700 m2 room, consumes 21 megawatts of electricity and must be cooled by 350 pumps that inject more than 22 m3 of water per minute.

But it is its speed that has the international HPC (High Performance Computing) community buzzing: in one second, it performs 1.1 billion billion operations (addition, subtraction, multiplication or division of 20-digit numbers). This is more than the mythical barrier of the exaflop (1 billion billion calculations/second), which it is the first to officially break.

State-of-the-art hardware architecture

"Every two or three years, a new supercomputer makes it to the top of the TOP500, explains Brice Goglin, director of research in the TADaaM project-team at the Inria center at the university of Bordeaux. It's rarely due to a hardware revolution. Sometimes it's just a clever mix of processors, graphics cards and networks not yet commercially available." Important note: some states may have more powerful supercomputers... but keep their existence a secret.

Why is Frontier so fast? First, because it uses the latest state-of-the-art computing processors and graphics cards. Second, because the 145 kilometers of interconnecting cables between its 9,400 servers (!) make up a network that's cleverly designed to speed up data transfers. Finally, in these servers, the connections between processors have been optimized. "This is the key to performance, explains Brice Goglin. The bottleneck is not the computation itself, but the time it takes for the data to reach us."

hwloc from Inria, a standard in the HPC world

In addition, Frontier benefits from the latest version of Inria's hwloc software, which was created in 2009 by researchers at the Inria center at the university of Bordeaux and has become a standard in the HPC world. "hwloc describes the hardware architecture of the machine and physically locates each resource: processor, graphics card, memory, etc. This allows us to distribute the parallel computing tasks of a program in the best possible way. For example, if two tasks need to exchange a lot of data, they are executed in processors located close to each other."

hwloc works a bit like suggesting to a general, before a battle, an optimal distribution and mode of communication between ground troops, artillery and aviation. The difference is that it is not about leading an army, but about executing a highly complex computer program. By the way, the designers of this program do not have to rewrite it every time they change supercomputers: hwloc does the work for them!

© Inria / Photo H. Raguet

Six months of work to adapt to Frontier

To develop version 2.8, which propels Frontier to the top of the world rankings, Brice Goglin exchanged countless emails with AMD and HPE Cray, the industry groups that designed the supercomputer, over a period of six months. "hwloc is open source software and we explain the reasons for every change in its code. But vendors bring in the latest technology. And sometimes they couldn't disclose certain details yet."

Still, it was essential to know these trade secrets. In particular, to describe how the processors (capable of all kinds of calculations), graphics cards (very fast for simple and repetitive operations) and network cards (which send data to other servers) are connected in the 9,400 servers.

Beyond the exaflop wall, exascale applications

Indeed, the speed of data flow is so decisive that it relies on highly specialized interconnections. "It's as if instead of supplying a factory with water via a single point of entry, we connect the fire system, air conditioning, sanitary facilities and production equipment separately to the public network. The installation is more complex, but the water gets to where it's needed faster."

Frontier, which takes this idea to the extreme, needed to be finely modeled to better distribute tasks and data. The designers at hwloc rose to the challenge, and the result is a new world speed record.

In a few months, the supercomputer will be open to the scientific community. Breaking the exaflop barrier makes previously unthinkable calculations possible. The next step will be exascale, i.e. achieving this level of performance for complex applications: validating the fundamental laws of nature, simulating the functioning of the brain, analyzing the structure of molecules and proteins, studying nuclear fusion and fission, analyzing the mechanisms of climate change, and so on.

"Frontier compute nodes use an innovative way to attach the AMD Instinct™ MI250X accelerators and NICs to the optimized 3rd Gen AMD EPYC™ processor. HPC applications must precisely know how these components are connected to make the most of their performance. For instance, each Instinct accelerator gets much higher network bandwidth if it communicates through its local NIC, but this was difficult to achieve with previous software stacks. To support an easier implementation of this capability, we worked with hwloc developers to properly expose this topology information so that applications and runtime software could be optimized accordingly."

-- Bill Brantley, Fellow Design Engineer, AMD


More information