St. Paul, MN 55144 USA. M. Hodes is with the Department of Mechanical Engineering, Massachusetts. Institute of Technology, Cambridge, MA 02139 USA.
502
IEEE TRANSACTIONS ON COMPONENTS, PACKAGING, AND MANUFACTURING TECHNOLOGY-PART A, VOL. 18, NO. 3, SEPTEMBER 1995
Gas-Assisted Evaporative Cooling of High Density Electronic Modules Avram Bar-Cohen, Fellow, IEEE, Greg Sherwood, Marc Hodes, and Gary Solbreken
Abstract-Gas-assisted evaporative cooling (GAEC), its advan- high velocity liquid jets of FC-77, at an inlet temperature of tages in thermal packaging of microelectronics,and its implemen- 14.4OC, to cool 40 W, 0.65 x 0.65 cm ECL chips to average tation in a prototype high-performance computer module, are and peak chip temperatures of 64 and 72”C, respectively described. Attention is then turned to theoretical considerations in the flow of gas-liquid-vapor mixtures in narrow, parallel plate [2]. The SS-I system, thus, re-defined the state-of-the-art in channels, and to the design and operation of an appropriate thermal management capability, increasing both the surface experimental apparatus. Next, experimental results for the wall heat flux and the volumetric heat density, relative to the temperature, heat-transfer coefficients, and pressure drops are 2nd generation multichip modules in use in the early 1990’s presented and compared to theoretical predictions. It is shown [l], and lent new urgency to achieving further improvements that this novel technique is capable of providing effective thermal management for electronic components, in the presence of both in high density thermal management. Continued reliance on high heat fluxes (of order 20 W/cm2) and high thermal density conventional packaging for the advanced chips and architectures soon to become available will considerably dilute their (of order 50 W/cm3).
potential contributions to system level gains. A recently patented thermal management technique [3], using high velocity flow of a liquid-gas mixture in the narrow channels between populated substrates, appears to I. INTRODUCTION offer the possibility of further improvements in volumetric ECENT developments in the electronic industry point to thermal management capability. A prototype, high packaging the importance of three-dimensional packaging in high- density module, relying on this approach, has been sucend computers and other microelectronic products. The close cessfully operated by Cray Computer Corporation and has component proximity required to reduce transmission delays provided a formidable cooling capability, combining heat flux in “serial” architectures, as well as the large number of com- as high as 20 W/cm2, on the surface of each GaAs chip, ponents needed in architectures involving parallel operation with a volumetric density of 18 W/cm3 (six times the value of many processors, have lead to the emergence of very high attained in the SS-1) in a compact, 65 cm3, 1024-chip module. density packaging and high volumetric heat removal rates. The While the Cray-3 prototype, thus established the viability heavily parallel computers developed by Intel and Kendall of this thermal management approach for actual high flux Associates, as well as the Cray-3, Cray Research’s T3D, and chips, little attention was devoted to the parameters needed the SS- 1 by Supercomputer Systems Inc., all provide evidence to facilitate extrapolation of these results to other geomeof this trend. It is anticipated that by the latter half of this tries. Consequently, it was decided to undertake a detailed decade, thermal designers may well need to contend with as experimental and theoretical study of the complex thermal much as 500-W dissipation from a single chip, at a chip heat transport phenomena occurring in GAEC, focusing on the flux of nearly 100 W/cm2, a substrate/board flux of 25 W/cm2, extraordinarily high volumetric cooling rates associated with and module heat density approaching 10 W/cm3 [l]. this approach. This paper presents a description and discussion The thermal characteristics of the prototype SS- 1, completed of the pre-commercial implementation, as well as exploratory and tested in late 1992, lend credence to these trends and research, of this novel cooling technique. This paper will begin projections. The 22.8 x 16.2 x 35.3 cm, 27.3 kW CPU with a description of the GAEC approach, its advantages in “brick,” or module, provided a heat-removal capability of thermal packaging of microelectronics, and its implementation 15.4 W/cm2 on the substrate and 2.1 to 3.2 W/cm3 in the in a prototype high-performance computer module. Attention “brick.” By interspersing liquid-impingement plates between will then be turned to theoretical considerations in the flow of populated substrates, the system designers were able to use gas-liquid-vapor mixtures in narrow, parallel plate channels Manuscript received May 1994; revised March 1995. This paper was and to the design and operation of an appropriate experimental presented at the Intersociety Conference on Thermal Phenomena, Washington, apparatus. Next, experimental results for the wall temperature, DC, May 5-7, 1994. heat transfer coefficients, and pressure drops will be presented A. Bar-Cohen and G. Solbreken are with the Department of Mechanical and compared to theoretical predictions. Engineering, Thermodynamics and Heat Transfer Division, University of
Index Terms-Cooling, evaporation, immersion, heat flux, electronics.
R
Minnesota, Minneapolis, MN 55455 USA. G. Sherwood is with the Specialty Chemicals Division, 3M Corporation, St. Paul, MN 55144 USA. M. Hodes is with the Department of Mechanical Engineering, Massachusetts Institute of Technology, Cambridge, MA 02139 USA. IEEE Log Number 9413709.
A . GAEC in the Cray-3 Introduction
The Cray-3 was a pioneering effort, by Cray Computer Corporation, to employ gallium arsenide (GaAs) as the primary
107&9886/95$04.00 0 1995 IEEE
I
503
BAR-COHEN et al.: GAS-ASSISTED EVAPORATIVE COOLING OF HIGH DENSITY ELECJRONIC MODULES
Coolant Path
Optimum flow path of gas and liquid
PowerAogic Distribution Planes Circuit Boards
7 Protective Overlay
0.79 0.89mm GaAs Chips
10.2 x 10.2 cm Module contains 1024 GaAs chips, 15000 twist pins
GasNapor Out
- Twist Pin Interconnect
Fig. 1. Cray-3 module assembly.
FC-72 out (Bottom of Tank)
semiconductor material in a supercomputer [3]. Increasing the circuit speed with GaAs and reducing signal path length Fig. 2. Cray-3 CPU tank assembly. through high density packaging, provided fast clock speeds and shorter propagation delays, but also resulted in very high power densities. To deal with the thermal challenge posed were specified to average 25OC (+/-8OC) during operation of by this power density, a novel cooling technique, utilizing the module. Total power dissipation for a 16 processor, Craydielectric liquid contact cooling--combined with an inert gas 3, comprised of 80 logic modules and 256 memory module, flow-was designed and patented [3]. Helium was chosen was expected to approach 270 kW, including approximately as the “propellant” gas due to its high thermal conductivity, 90 kW in the power supplies. compatibility with the electronic components, and some early indication of reduced viscosity for helium-saturated FC. The C. Two-Phase Coolina high dielectric strength and chemical inertness of 3M’s FluThe cooling system developed for the Cray-3, shown in orinert liquids, and the desire for a low boiling point, made Fig. 2, was designed to: 1) minimize the mass flow of coolant FC-72 the natural coolant choice. The low surface tension, low through the module, thus reducing stress on its delicate strucviscosity, and high density of this liquid could also be expected ture, and 2) maintain a uniform temperature across the module. to assist in the helium-driven distribution of the fluid through The desire to reduce liquid mass flow rate led to consideration the small channels and clearances in the module. This thermal of evaporative cooling. By distributing FC-72 with a nozzle management technique was implemented in a prototype Cray-3 over the top of the module and pressurizing the volume above module and found to offer outstanding cooling capability. with an inert gas, a unique flow configuration was established, as the liquidgas mixture flowed over the GaAs chips. The B. Prototype Module module was oriented vertically, relying on gravity and a The Cray-3 prototype module, shown in cross-sectional modest pressure differential of 20.6 kPa (3 psi) to circulate view in Fig. 1, contained 16 stacks of 4 circuit boards, the coolant. each populated by 16 GaAs flip-chips, 3.9 x 3.9 mm, and In early visualization studies, with a transparent “layer” of dissipating up to 3 W, for a peak chip heat flux of nearly the module, it became apparent that careful sealing of the 20 W/cm2 and an average board heat flux of 7.7 Wlcm’. A module was required to assure vertical flow of the coolant powerllogic distribution plate served as the backbone of the through the channels between boards. Flow restricting commodule and stranded wire “twist pins” provided for signal and ponents were designed into the YO wire net and the power power interconnect between layers of the module. A fully- connector was redesigned to provide a liquid-tight seal to the assembled circuit board has a profile thickness of a nominal power bus. These design features are covered under the cooling 0.89 mm. Each board in the stack is separated by 0.51 mm system patent [3] and were essential in maintaining proper spacers, providing a clearance of 0.1 mm from the back of each coolant flow over the chips. GaAs chip to the next circuit board. This design provided for In the Cray-3, the fluid and the gas were circulated in an extremely high power density, yielding as much as 1.5 kW independent loops. The FC-72 was pumped from a tank per logic module, of dimensions 10.1 x 10.1 x 0.64 cm (0.3 assembly, sprayed over the modules, carried through with kW per memory module), and approaching 18 W/cm3 (300 the helium, and drained into a liquid reservoir, where the W/in3) in several sections of the system. Chip temperatures power supplies were immersed in this dielectric liquid. The Y
504
IEEE TRANSACTIONS ON COMPONENTS, PACKAGING, AND MANUFACTURING TECHNOLOGY-PART A, VOL. 18, NO. 3, SEPTEMBER 1995
gashapor mixture was drawn out of the volume, above the power supplies (maintained at approximately 6.9 kPa gage), by a positive displacement blower. A heat exchanger following the blower, served to remove the heat of compression and to condense the vaporized FC-72. A vortex separator was used to trap the liquid and return it to the low pressure side of the liquid circulation pump. The gas passed through a particulate filter and then returned to the volume above the modules at a nominal gage pressure of 27.5 E a . To assist in the development of this cooling system, special modules--containing chips with embedded thermal diodes-were tested, under realistic thermal conditions. Each module was supplied with 1.9 Ymin (0.5 gaYmin) of 14°C temperature FC-72 and 0.47 V s (1 CFM) of 14OC helium. Chip temperatures in a fully-powered module were found to lie in the range of 25OC to 3OoC and heat transfer coefficients on the chips were estimated to be as high as 3 to 4 times better than all liquid flow. Due, however, to the complex thermofluid interaction of the liquid and gas in the mixing chambers and narrow channels within the modules, it was difficult to more precisely quantify the thermal performance of this GAEC system. Consequently, a laboratory apparatus, consisting of a single, asymmetrically heated channel, was designed and built at the University of Minnesota’s Laboratory for the Thermal Management of Electronics, to study the details of heat transfer and fluid flow in this cooling technique. Due to environmental considerations and the desire to focus on the hydrodynamic effects of the “propellant” gas, in these laboratory experiments, nitrogen gas was used instead of helium.
100 10 1
0.1 0.01
, I
I
0.001 0.1
10
1
100
UGS[ d s l Fig. 3. Two-phase flow regime map generated using .Vz and FC-72 (liquid) properties. Inlet Conditions-All liquid: L;I,S = 0.18 d s , I’CS = 0.0 d s . (A) LTLS z 0.18 ds, IT
