The Wonderful World Of Liquid Cooling

By Michael McNerney, Supermicro

In the last article, we discussed the rise in liquid cooling and why it is so important to today’s data centers. This piece will focus on the details about each type of liquid cooling, the benefits and drawbacks of each, and provide some initial guidance as to which option could suit an enterprises’ needs.

There are three main options for implementing liquid cooling technology: direct-to-chip, immersion, and rear door heat exchangers. Similar to the very choice between air and liquid cooling for a data center we discussed in part one– each specific deployment and equipment has different needs that may be better suited to a certain liquid cooling approach.

Direct-To-Chip Cooling

The first is to utilize direct-to-chip (D2C) cooling technology. This is the liquid cooling approach most are familiar with – as it is what has been adopted by the PC enthusiast community. A D2C cooling setup allows for cold liquid to flow over the heat-generating electronics so that the liquid can absorb the heat away from the chips themselves. The warm liquid is then pumped into a cooling system, using airflow and fan cooling to bring the warmed liquid back down to a designated cool temperature. This cooled liquid then continues to flow back toward the hot electronics to continue this cooling cycle.

Direct-to-chip is a very reliable form of cooling, however, the setup can be an extensive one. Most D2C setups have multiple liquid “loops”, pipes where the cold liquid is pumped to a specific part of the computer – like the CPU, GPU, and/or memory – to cool it. With a server rack that has dozens of CPUs, GPUs, and memory sockets that need to be cooled, having multiple loops can be a necessity. Hence, many companies opt to create water liquid distribution blocks, similar to a PDU in rack distribution. This allows for multiple loops to be created and connected more easily.

Because D2C liquid cooling is still dependent on air-cooling, it’s not the most efficient option available – only having a power to cooling efficiency of about 70%. Hence why many manufacturers explore more extreme forms of liquid cooling.

Immersion Tank Cooling

The most direct form of cooling after D2C, which is even more efficient, is immersion cooling. Immersion cooling is when the entire server is immersed in a tank containing a non-conductive liquid. As the server lets off heat the temperature of the surrounding liquid rises and the liquid itself physically rises, while cool liquid is pumped into the tank from the bottom. The warmer liquid is then pumped out of the tank to be cooled externally. Because immersion cooling utilized a closed loop, not only does it not need any fans but it has been proven to be extremely efficient at moving away the heat generated by CPUs and GPUs.

The immersion approach also has the added benefit of improving the reliability and lifespan of hardware, because there is no exposure to oxygen and other corrosive materials. This is a particular benefit for servers and data centers that need to operate at high elevations, according to 2018 research by Alibaba. Because of the changes of air density at elevation, air cooling tends to be less effective – and in fact, immersion tanks have the strongest level of cooling success at elevation in comparison to all other types.

However, this doesn’t mean the immersion tanks are the optimal cooling solution. The biggest challenge with trying to implement immersion cooling is the historical design of servers and data centers. Standard facilities are currently built for standing racks, not a vertical tank. Immersion tanks require ceiling hoists to be installed, as servers have to be lifted vertically out of the tank, which is an expensive installation that some facilities simply aren’t equipped for.

Furthermore, since immersion tanks are filled with a non-conductive liquid, servicing the immersed servers can be troublesome. The servers need to be completely removed from the tanks and cleaned of all fluid before technicians can operate on them. Most experts thus recommend hiring an immersion cooling team to help your company integrate into immersion cooling. While this is an investment by the data center, the results can significantly reduce the overall power usage and costs for a data center.

Rear Door Heat Exchangers

While not technically a liquid cooling system, another popular method to remove heat closer to the source is to install Rear Door Heat Exchangers (RDHx). These are attachments fitting the back of your current standing rack, containing a series of pipes that have cool liquid running through them. By cooling the air at the back of the servers (the exhaust) with these doors, the racks themselves can be cooled much more effectively with standard fan cooling.

This approach allows data centers to keep their current air conditioning systems but operate them at a lower intensity. The RDHx also can be attached to traditional vertical racks and servers, so there is no need to replace equipment or redesign the facility itself. Additionally, many RDHx come with moisture detection sensor that allows servers to automatically shut down off automatically if there are any leaks, offering a stronger level of safety for operators. Overall, this makes this approach much less expensive and easier for many businesses to consider for a liquid cooling option.

Conclusion

Each business and data center has unique needs, and there is a need to find the perfect cooling match. Many different options exist that will allow the next generation of systems to operate at maximum performance while reducing overall data center energy consumption. The overall reduction in the Power Usage Effectiveness for servers using liquid cooling is compelling. Data centers leveraging various liquid cooling technologies can bring PUE’s down into the 1.1 to 1.25 range, optimizing compute performance for power utilized and saving a significant amount of OPEX costs.

About The Author

Michael McNerney is VP of Marketing and Network Security at Supermicro. Michael has over two decades of experience working in the enterprise hardware industry, with a proven track record of leading product strategy and software design. Prior to Supermicro, he also held leadership roles at Sun Microsystems and Hewlett-Packard.