Thought Leader Thursday: Trends in HPC – Exascale

Share This

If you’re like me, you probably love predictions about the future of technology. The most thought-provoking predictions promise radical changes in infrastructure, business, and culture… basically, completely changing life as we know it. However, if you are into technology, you are probably also a skeptic (I am!), and you know that many of these “predictions” are what I call “push predictions” (after “push polls”) intended to sway opinion and move the market in a particular direction.

So rather than looking forward, I wanted to begin this first “Trends in HPC” post by taking a look back (predicting the past is much easier than predicting the future). Here is my rough view of the HPC landscape over the last several decades, looking at key aspects of Systems, Data, and Business.


Clearly, there are lots of interesting trends across the decades. One of my favorites is how history repeats itself — e.g., the 8087 floating point co-processor introduced in 1980 was quickly integrated into the CPU and the same is happening with today’s accelerators and co-processors.

In this post I’d like to discuss the ever-increasing scale of HPC systems. This scalability trend is currently focused around Exascale, computing at speeds one hundred times faster than the fastest computers that exist today. “Exascale” is derived from ExaFLOPS (“exa” as in 1,000,000,000,000,000,000 and “FLOPS” as in Floating-point Operations Per Second).

Exascale infrastructure will require advances in four areas: scale, speed, resilience, and power management. From the trends I mentioned before, the need to support ever-increasing scale and to work around ever-decreasing resilience is clear:


Actually, scale, speed, resilience and power management are really interdependent. With greater scale comes the need for greater software efficiency (speed) and greater power. The resilience connection comes from the need to keep costs affordable: as systems have increased in scale, the industry has moved from using a small set of proprietary (expensive) components to using a very large set of commodity (inexpensive) components. The thing about inexpensive commodity components is that they are also more unreliable. (That’s one reason they are less expensive!)

Altair is playing an important role here, both by scaling up our HyperWorks solvers to provide accurate, repeatable results at huge scale, and by providing PBS Works (my main area of focus). In particular, PBS Professional provides job scheduling and workload management – “must have” core capabilities for every HPC system – ensuring HPC goals are met by enforcing site-specific use policies, enabling users to focus on science and engineering rather than IT, and optimizing utilization of hardware, licenses, and power to minimize waste.

We just released PBS Pro 13.0, and we’re confident the underlying architecture is the right foundation in all four Exascale areas:

  • The new TPP layer provides exceptional scale (see Techie Details below);
  • Performance improvements, including a new asynchronous job startup, provides a 10x improvement in speed;
  • The new 100% health check framework provides the basis for exceptional resilience; and
  • Concurrent with the PBS Pro 13.0 release, we have also released new power management facilities for SGI and Cray systems.

One final thought: the focus on bigger and faster doesn’t just benefit those with massive computing infrastructures; everyone benefits from faster, more resilient infrastructure – the small, the medium, and the really big.

trends hpc

Techie Details: PBS Pro 13.0 “Architected for Exascale”

When PBS Pro was originally designed more than 20 years ago, we built our own PBS-to-PBS, scalable communication layer. As the TCP/IP stack was pretty immature (both fragile and not so speedy), we built a reliable packet protocol based on UDP called “RPP”. Back then, individual computers were very expensive, so trading a little reduction in speed for an increase in reliability was important. RPP has served PBS Pro amazingly well, but it was reaching its limits, so we embarked on a multi-year project to completely replace it.

Three fundamental changes have taken place in the industry since we developed RPP: system (cluster) sizes have gotten much larger; individual computers have become much less expensive; and “the web” changed the fabric of the internet, leading to great improvements in the TCP/IP stack (which now easily handles tens of thousands of connections on a single socket).

The new PBS-PBS communication layer (“TPP”) leverages these fundamental changes. Our “TPP” has been built from the ground up; it uses multi-threaded (non-blocking) messaging for improved speed, persistent connections to reduce latency, and it is fully fault resilient to ensure node/link failures do not impact operation.

Have more questions about the technical details? Please contact Altair and we will be happy to follow up.

Bill Nitzberg
Latest posts by Bill Nitzberg (see all)

Share This
Bill Nitzberg

About Bill Nitzberg

Dr. Bill Nitzberg is the CTO of PBS Works at Altair Engineering, Inc. With over 30 years in the computer industry, spanning commercial software development to high-performance computing research, Dr. Nitzberg is an internationally recognized expert in parallel and distributed computing. Dr. Nitzberg has served on the board of the Open Grid Forum, he co-architected NASA’s Information Power Grid, edited the MPI-2 I/O standard, and has published numerous papers on distributed shared memory, parallel I/O, PC clustering, job scheduling, and cloud computing. In his spare time, Bill tries to reduce his pack weight for his long-distance hiking trips.

Comments are closed.