Immersion cooling : a promising cooling technology for servers, as long as you master it

What is immersion cooling?

Immersion cooling is when the electronic components of servers are completely submerged in a dielectric liquid (a mineral oil to avoid short circuits and oxidation) to cool them down. The servers are cooled by the circulation of the oil through convection (either naturally or forced by a pump), which limits the need to use energy-intensive air conditioning systems. What makes the system all the more effective is that the oil is an excellent conductor of heat (much better than air).

In concrete terms, the heat produced by the electronic components of the servers (CPUs, GPUs, etc.) is transferred to the oil, which in turn transfers this heat into a water circuit via a heat exchanger. The temperatures of the water circuits are much higher than in traditional data centers, and the need for air conditioning is significantly reduced.

Air coolers, often adiabatic (by evaporation), make it possible to expel this heat directly into the air outside the data center. Even better, the heat from the water circuit can be used to heat homes, offices, or even a swimming pool. Finally, the absence of fans on the servers makes them use less energy too.

What situations is immersion cooling best suited to?

In data centers, the main use case for immersion cooling is a high-power-density situation; in other words, where you have high amounts of processing power in a small space. This is the case, for example, with solutions for artificial intelligence (AI) and online games, as well as all data centers with HPC processors (research laboratories, aerospace, some specific activities in finance, health care, etc.

Some bays can now house 50 kW, 100 kW, or even more. In this context, immersion cooling should enable a drastic reduction of electrical energy consumption: the synthetic oils used are thought to be capable of absorbing 1,500 times more thermal energy than air at the same volumes and temperatures.

What’s more, the first environmental studies (simplified life cycle assessments) carried out at the Lab-by-APL using the NegaOctet database applied to a theoretical scenario show that immersion cooling with a silicone-based dielectric liquid generates between 30% and 50% less CO2 emissions than traditional cooling (direct expansion) over the complete life cycle (manufacturing, transportation, use, and end-of-life) of the equipment and the liquid.

What types of equipment are compatible with immersion cooling?

Not all existing server racks are compatible with an immersion solution: for one thing, they aren’t necessarily the right shape to fit into the immersion cooling units (rectangular oil baths) and, for another, the design of traditional servers doesn’t allow the oil to circulate.

In the second case, modifications are necessary, like the removal of fans or the replacement of certain materials, such as thermal paste, which has to be replaced by another component since it will dissolve in any dielectric liquid.
As for hard drives, while SSDs are natively compatible with immersion cooling, traditional rotary hard drives are not, unless they have been designed to be hermetically sealed or certified by the manufacturer.

“Basic” optical fiber is generally not compatible with immersion. To be able to use fiber connections, it is necessary to use specific cable covers.

However, several manufacturers are already starting to offer immersion-ready servers in their catalogs.

Are there situations where immersion cooling might not be suitable?

That is mainly the case for data centers that need a very high degree of agility, because immersion cooling systems can be less mobile than traditional cooling infrastructure. This can apply, for example, to infrastructure or cloud providers, who need to quickly and regularly change the “small” servers. This is not necessarily possible with immersed servers, due to the precautions necessary when handling them.

Moreover, the maintenance of immersion cooling systems seems to be more difficult because of the need to handle dielectric liquid and maintain the integrity of the sealed systems. Trained staff are needed to maintain this type of equipment, and may require more time and equipment (leak trays, cloths and gloves, lab coats, service carts, etc.).

In addition, while certain dielectric liquids are designed to minimize corrosion, an in-depth assessment of the equipment is always recommended before implementing an immersion cooling solution.

Finally, there are not yet any norms or standards governing immersion cooling.

Does immersion cooling mark the end of data center air cooling?

The performance of immersion cooling makes it very attractive from a theoretical point of view, particularly for high-density data centers. However, this technology does have downsides, mainly in relation to operation and maintenance, which make it more suitable for certain specific use cases.  For now, air cooling remains a valid alternative that has its place in traditional applications.

Moreover, there have been few in-depth (technical-economic) studies or reports on the successful use of immersion cooling on a large scale. A detailed assessment should therefore be carried out on the risks and how they can be mitigated before implementing this solution.

Research and innovation are continuing and standardization initiatives have been launched, like the one by the OCP (Open Compute Project). In the long term, these initiatives should enable the data center sector to master the design, operation, and maintenance of immersion cooling infrastructure from end to end.