US10123463B2 — Liquid submerged, horizontal computer server rack and systems and method of cooling such a server rack (Part 1 of 2)
Document text
Research, not advice. Part of the Bitcoin research archive (October 2026). Claims labelled unverified, contested or fringe are reported, not endorsed; statuses of bills and rules are as of the date checked. Government, court and patent records are public domain; the research notes are CC BY 4.0.
US010123463B2
(12) United States Patent (10) Patent No.: US 10 ,123,463 B2
Best et al. (45) Date of Patent: Nov. 6 , 2018
(54 ) LIQUID SUBMERGED , HORIZONTAL 2) U . S . CI.
COMPUTER SERVER RACK AND SYSTEMS CPC ......... HO5K 7 /20772 (2013 .01 ); F28D 15 /00
AND METHOD OF COOLING SUCH A ( 2013 .01 ); G06F 1/20 (2013.01);
SERVER RACK (Continued )
(75 ) Inventors: Christiaan Scott Best , Austin , TX ( 58 ) Field of Classification Search
(US ); Mark Garnett, Oklahoma City,
OK (US)
(58)Fipa
CPC .
...
....
....7120636
.. HO5K;7 / 20872 , H05K 7 /20627 ; HO5K
7 /20636 ; HO5K 7/20645 ; HO5K 7 /20654;
(Continued )
( 73 ) Assignee : Green Revolution Cooling , Inc.,
Austin , TX (US ) (56 ) References Cited
( * ) Notice: Subject to any disclaimer, the term of this U . S. PATENT DOCUMENTS
patent is extended or adjusted under 35 4 ,590 ,538 A * 5/ 1986 Cray, Jr. ........... 361/700
U .S .C . 154 (b ) by 971 days . 4 ,834 ,257 A * 5/ 1989 Book et al. .................. 220 /646
13/057,881 (Continued )
(21) Appl. No.:
FOREIGN PATENT DOCUMENTS
( 22 ) PCT Filed : Aug . 10 , 2009
101443724 5 /2009
(86 ) PCT No.: PCT/US2009 /053305 2004319628 11 /2004
$ 371 (c )( 1), (Continued )
( 2 ), ( 4 ) Date : Feb . 7, 2011 OTHER PUBLICATIONS
(87) PCT Pub . No.: W02010/019517 Inetl Core 2 Duo Processor on 65 nm process for Embedded
PCT Pub . Date : Feb . 18 , 2010 Applications, Aug . 2007, Intel, pp . 1, 14 , and 21 . *
(65) (Continued )
Prior Publication Data
US 2011/0132579 A1 Jun . 9, 2011 Primary Examiner — Christopher R Zerphey
Related U .S . Application Data (74 ) Attorney, Agent, or Firm — The Marbury Law
Group , PLLC
(60 ) Provisional application No.61/ 188 ,589 , filed on Aug . ABSTRACT
11, 2008, provisional application No. 61/ 163, 443 , (57 )
(Continued ) Apparatus , systems, and methods for efficiently cooling
computing devices having heat-generating electronic com
(51) Int. CI. ponents, such as , for example , independently operable serv
F28D 15 /00 ( 2006 . 01) ers , immersed in a dielectric liquid coolant in a tank .
H05K 7 /20 ( 2006 .01)
G06F 1 /20 ( 2006 .01) 33 Claims, 14 Drawing Sheets
844
P 874 876
- HEATED LIQUID COOLANT
820 860
830
COOLED LIQUID COOLANT
880860 862872
US 10 ,Page
123 ,2463 B2
Related U .S . Application Data 2008 / 0029250 A1 * 2/ 2008 Carlson ............... F24F 11/ 0001
165/ 104 . 33
filed on Mar. 25, 2009, provisional application No. 2008 /0030945 A1* 2 /2008 Mojaver et al. .............. 361/685
61/ 165,470 , filed on Mar. 31, 2009 . 2008 /0055845 AL 3 /2008 Murakami
2008 /0158818 AL 7 /2008 Clidaras
2008/0196870 A1 8 /2008 Attlesey
(52) U .S . Ci. 2009 /0260777 AL 10 /2009 Attlesey
CPC ............... GOOF 1/206 ( 2013 .01) ; H05K 7 /20 2010 /0226094 A1 * 9 /2010 Attlesey et al. .. .......... 361/699
(2013 .01); H05K 7/ 203 (2013 .01 ); H05K 2010 / 0246118 AL 9 /2010 Attlesey
7 / 2079 (2013 .01 ); H05K 7 /20236 (2013 .01) ; 2010 /0290190 A1 11/2010 Chester et al.
H05K 7 /20281 ( 2013 .01) ; H05K 7/ 20327 2010 /0302678 Al 12 /2010 Merrow
2011/0075353 AL 3 /2011 Attlesey
(2013 .01 ); H05K 7 /20381 (2013 .01 ); H05K 2011 /0132579 AL 6 / 2011 Best et al.
7 /20763 ( 2013 .01 ); H05K 7 /20781 ( 2013 .01) ; 2011 /0240281 Al 10 /2011 Avery
H05K 7 /20827 ( 2013 .01) ; H05K 7 / 20836
( 2013 .01) ; G06F 2200 /201 (2013 .01 ); HOIL FOREIGN PATENT DOCUMENTS
2924 /0002 (2013 .01) ; YIOT 29/4973 (2015 .01 ) JP 2004363308 12 / 2004
(58 ) Field of Classification Search WO WO 2007023130 3 /2007
CPC ........... HO5K 7 /20236 ; H05K 7 /20709; HO5K WO WO 2007098078 8 / 2007
7 / 20218 ; HO5K 7 /20763 ; HO5K 7 / 203 ; WO WO 2008027931 3 /2008
HO5K 7 / 20781 ; HO5K 7 / 2079; G06F Wo WO 2008089322 7 / 2008
1 / 20 ; G06F 2200 /201; HO1L 23 /473 ; WO 2010019517 2 /2010
F24F 1/ 02 , F25D 17 /02
USPC .............. 62/ 259. 2, 263, 434 , 435 ; 361/699 ; OTHER PUBLICATIONS
165/ 104 .33
See application file for complete search history . International Search Report and Written Opinion dated Oct. 14 ,
2009, Application No. PCT/US2009 /053305, 10 pages.
(56 ) References Cited Singapore Written Opinion and Search Report dated May 2 , 2012 ,
Application No. 201100595 -6 , 21 pages .
U . S . PATENT DOCUMENTS Examination Report from Australian Application No. 2009282170,
dated Nov. 15 , 2013 , pp . 1- 3 .
5 ,297,621 A 3/ 1994 Taraci Patent Examination Report No . 2 from Australian Application No .
6 , 374 ,627 B1 * 4 / 2002 Schumacher et al. ....... 62/259 . 2 2009282170 , dated Jun . 18 , 2014 , pp . 1-4 .
6 ,600 .656 B1 * 7 /2003 Mori et al. .................... 361/724 International Search Report and Written Opinion from PCT/US12 /
6 ,621, 707 B2 9 / 2003 Ishimine 49668 , dated Oct. 19 , 2012 , Green Revolution Cooling Inc., pp .
6 ,909,606 B2 6 / 2005 Barsum 1 - 10 .
7 ,086 ,247 B2 8/ 2006 Campbell Office Action from Chinese Application No . 200980131707.3 , dated
7 ,184 ,269 B2 * 2 /2007 Campbell et al. .......... 361/700 Dec . 31, 2014 , English and Chinese versions, pp. 1- 10 .
7 ,210 , 304 B2 5 /2007 Nagashima Office Action from Chinese Application No. 200980131707.3 , dated
7 ,307, 841 B2 12 / 2007 Berlin Apr. 3 , 2014 , English translation , pp . 1 - 13 .
7 ,318 , 322 B2 * 1/ 2008 Ota et al. .................. 62 /259. 2 Office Action from Chinese Application No. 200980131707.3 , dated
7 ,403, 392 B2 7 / 2008 Attlesey
7 ,609 ,518 B2 10 / 2009 Hopton Dec. 20 , 2012 , English and Chinese versions, pp . 1 - 17.
7 , 905, 106 B2 3 /2011 Attlesey Office Action from Chinese Application No . 200980131707 .3 , dated
7 ,911,782 B2 3 /2011 Attlesey Jul. 31 , 2013 , English and Chinese versions, pp . 1 -8 .
7 ,911, 793 B2 3 /2011 Attlesey Office Action from Chinese Application No. 200980131707 .3 , dated
8 ,009 ,419 B2 8 / 2011 Attlesey Apr. 18 , 2014 , English and Chinese versions, pp . 1 - 7 .
2002/0185262 Al 12 / 2002 Baer Office Action from Chinese Application No. 200980131707 .3 , dated
2003/ 0053293 Al * 3/ 2003 Beitelmal et al . ............ 361/687 Jul. 31, 2013, English translation , pp . 1- 3 .
2003 /0127240 A17 /2003 Beckbissinger U . S . Appl. No. 14 /338 ,035 , filed Jul. 22, 2014 , Christiaan Scott
2004 /0008490 A1 * 1/ 2004 Cheon . .... 361/699 Best.
2004/0246683 A1 * 12/ 2004 Honsberg -Riedl et al. .. 361/720 U .S . Appl. No. 14 /338,013 , filed Jul. 22, 2014 , Christiaan Scott
2005 /0083657 AL 4 / 2005 Hamman
2005 /0114876 A 5 / 2005 Atarashi Best.
2005 /0259402 A1 * 11/ 2005 Yasui et al. ... ................ 361/716 U .S . Appl. No . 14 /338,020 , filed Jul. 22, 2014, Christiaan Scott
2006 / 0026610 A1 2 / 2006 Sasao Best.
2006 /0064709 Al 3/ 2006 Throckmorto U .S . Appl. No . 14 /338 ,026 , filed Jul. 22 , 2014 , Christiaan Scott
2006 /0123436 AL 6 / 2006 Tanaka Best.
2006 /0135042 A16 / 2006 Frost et al.
2006 /0250755 AL 11/2006 Tilton
U .S . Appl. No. 14 /667,091, filed Mar. 24 , 2015 , Christiaan Scott
Best .
2006 /0274501 A1 * 12 / 2006 Miller . 361/690 Canadian Office Action dated Nov . 8 , 2016 for Canadian Patent
2007 /0006599 A1 * 1/ 2007 Kawamura et al. ........... 62/ 54 . 1 Application No. 2 , 731, 994 related to National Phase of Application
2007/ 0025081 A1 * 2 / 2007 Berlin et al. .................. 361/698 No. PCT/US2009/053305 filed Aug . 10 , 2009 .
2007 /0034360 A1 * 2/ 2007 Hall .. 165/ 104 . 33 Non - Final Office Action in U . S . Appl. No . 14 /667,091 , dated May
2007 /0199340 A1 * 8 /2007 Knight et al. ............... 62/259. 2
2007/ 0213000 AL 9 / 2007 Day 22, 2015 , 16 pages.
2007/0267741 A1 * 11/ 2007 Attlesey et al. .............. 257 /714 Substantive Examination Adverse Report, Malaysian Application
2008 /0002364 A11 /2008 Campbell et al. No . P12011000494 , dated May 15 , 2015, 3 pages.
2008 /0017355 A1 * 1/ 2008 Attlesey et al. ......... 165 / 104 .33
2008 /0026509 A11 /2008 Campbell * cited by examiner
U . S . Patent Nov. 6 , 2018 Sheet 1 of 14 US 10 , 123, 463 B2
DLFIELQEUCOTIRIWDC
CFOLOWULINDG
100
r
INFORMATION
-
.
Fig
1A
120
110
.
1
140 150
1130 160
RARACK
120
PUMP FILTER HEAT EXCHANGER COLING APARATUS
170 180 CONTROLER
U . S . Patent Nov. 6 , 2018 Sheet 2 of 14 US 10 , 123, 463 B2
DLFIELQEUCOTIRIWDC
CFOLULOINWDG
200
-
INFORMATION
-
-
-
-
-
-
260
+
4280 250
210
120
RACK HEAT EXCHANGER 1290 COLINGAPARATUS Fig
1B
.
270
CONTROLER
U . S . Patent Nov . 6 , 2018 Sheet 3 of 14 US 10 , 123, 463 B2
300
370 -
1 , 310
RACK
-
-
TEMP
380 DATA
PUMP
CONTROLLER 340
FILTER
-
CONTROL 1 , 390
-
-
3 WAY
VALVE
-
-
-
-
-
-
-
356 350
-
OUTSIDE AIR REFRIGERATION HEAT
RADIATOR RECOVERY
------------ 354 - - - - - - - - - - - - - -
SIGNAL
- FLUID EVAPORATIVE
- -- - COOLER
- - - - - - - - - - -
- - - - - - -
358
Fig . 2
U . S . Patent Nov. 6 , 2018 Sheet 4 of 14 US 10 ,123 ,463 B2
400
400 120 000
Fig
4
.
I 120
460
120
|
472
et
TI
450 422 IUK
120
D
ww 440
442
Fig
3
.
430
.
410
440
U . S . Patent Nov . 6 , 2018 Sheet 5 of 14 US 10 , 123, 463 B2
H
FILL
472
400
5
.
Fig 120 -
-
-
-
-
-
-
-
H -
-
422
w7460 boy 120
+ l
Hi
420
460
120
-
-
.
6
Fig
450 HS4407 P
IU
-
-
-
-
120
-
-
-
*
400 K -
+
-
120
450 440
U . S . Patent Nov. 6 , 2018 Sheet 6 of 14 US 10,123 ,463 B2
22 $(3()
????????
?? ???? * * ? ????? - -
?3%
562/ ???????
??
096
????????????
????
?? ? ? ? ? ?:
?
r????
antyi???p
r nwmuw?uary
?
rt?r+
on????
; ?
?
LABLE
?
?
?563
???
???•??•?a
???? *
?? ?
* *
.
SS
Fig . 7
?
????
???????????????
? ? ?? ?
? ????? ? ? L L A N A R A L A R A L M
?
A L ?t
-
-
.
?
??r
- 56(0
?AAHAKARAARArKAAAAgAVA????AyAKAAEaRA?AALHAsAALApAAAExAR?RAALRLAMuRLAAALaANA?RRrAAALA?ARAAiLLAAALAML?AAANMABKLAAALNAA????otrMiLL?AAxMALA?tAMLAAAMAAALLALA?LAAAMtLLAAAMALAr*? ??????
MAKALARR?????
???
AKAL ARALAMIRAL EL ???,?s
???????????????????ntr?????????u???????????*r
*
ria r
???* * * * * * * * * * * * * * * * ?????????
?? ???t???
??
Fig . 8
U . S . Patent Nov. 6 , 2018 Sheet 7 of 14 US 10 , 123, 463 B2
620
600
660 622 674 610
tel
676
O
O
O
O
670
660
Fig. 9 660 672
720 776 722
710 v 774 l 770
760 760
750
710
Oooo
760 780 784 1782
Fig . 10
U . S . Patent Nov. 6 , 2018 Sheet 8 of 14 US 10 , 123, 463 B2
·
840
5810 RACK
844 3
T
11
.
Fig
832
872 -
830
800 #
820
U . S . Patent Nov. 6 , 2018 Sheet 9 of 14 US 10 , 123, 463 B2
820
-
1891 .
12B
Fig
12A
Fig
.
13
Fig
.
836
0842 47840
832
EFTIHEB/RNERT
830
8404
U . S . Patent Nov. 6 , 2018 Sheet 10 of 14 US 10 , 123, 463 B2
CHLOEIAQLTUAEINDT CLOIOQLUAEINDT
860
H
JA
876
8
72
874
862
844
e 860 Fig
14
.
830 880
836
8207
U . S . Patent Nov. 6 , 2018 Sheet 11 of 14 US 10 , 123, 463 B2
900
901
310 J7
315
--- 920
916 924
L- -
925
- - -
315
-
-
-
-
916 918
- - --- -
11
.
914 914" Tec1 908 Tu - 906 904 902 -
-
-
-
-
-
-
- FLUID LINE - -
-
- -
- - - - - CONTROL LINE
- - - - - - - - - - - - - - -
| 912 928
TCRA TH, R
MRA MR
Fig . 15
atent Nov . 6 , 2018 Sheet 12 of 14 US 10 , 123, 463 B2
10 6
120 22
DFTSLIHELORQEWOVHCUIETRNGIDHSC TMDLEOIMNEPILEQTFHROCUARTIUENRDGC DTEOLPTMETRVMHIAMNTIUENRMDGE DSENCTEROMHEVRDINGIRNLYDSG DSAMOCUEPTISNCEOARINMHLGRIDAMENTGIU1ZRNYMG8ES DTECLHIXOECUSLRHAQPTEOMCLTUANILGERNLITDGYRC AERTHDXEMJCRUHOEFBESACUTNIGNTHDGR TMEALMIEPNVHRTATUIERNDE 16
Fig
.
atent Nov . 6 , 2018 Sheet 13 of 14 US 10 , 123,463 B2
FLOWING DIELECTRIC LIQUID 24
AT A LOWER TEMPERATURE
INTO THE TANK WITH
SUBMERGED SERVERS
FLOWING THE DIELECTRIC 26
LIQUID THROUGH THE
SERVERS FOR ABSORBING
HEAT FROM THE SERVERS
OPTIONALLY ENHANCING 28
- - - - - - - - - - - - - - - -
- - - - - - - - - - - - -
CONTROLLER
FLUID VELOCITY OF
DIELECTRIC FLUID THROUGH
THE SERVERS APPLICATIONS
SOFTWARE
MONITORING THE TEMPERATURE OF
THE DIELECTRIC LIQUID AND SERVER -
-
_
_
ELECTRONIC COMPONENTS AND -
-
-
_
_
FLOW RATE OF THE DIELECTRIC LIQUID
_
-
_
-
_
-
_
-
_
-
_
-
SELECTING THE MORE 132 -
-
_
_
_
EFFICIENT SECONDARY -
L
COOLING SYSTEM TO -
-
-
MINIMIZE ENERGY USAGE -
-
-
-
-
PUMPING THE HEATED -
-
DIELECTRIC LIQUID TO A 34 -
-
DISTALLY LOCATED HEAT
EXCHANGER -
-
-
REJECTING THE ABSORBED | 36 -
-
HEAT IN THE DIELECTRIC FLUIDLY COUPLING THE
LIQUID AT THE HEAT COOLED DIELECTRIC LIQUID
EXCHANGER ASSOCIATED WITH TO THE TANK
CHOSEN SECONDARY
COOLING APPARATUS
ADJUSTING THE FLUID FLOW 140
OF THE SECONDARY
- -
Fig . 17A
COOLING APPARATUS
44
DISSIPATING THE REJECTED 42 RECOVERING THE REJECTED
HEAT IN A SECONDARY HEAT IN A SECONDARY
COOLING APPARATUS COOLING APPARATUS
U . S . Patent Nov . 6 , 2018 Sheet 14 of 14 US 10 , 123 ,463 B2
RECEIVE SIGNALS OF SYSTEM
OPERATION FROM SENSORS: 152
FLUID FLOW , POWER,
TEMPERATURE
DETERMINE OPTIMUM
ELEVATED TEMPERATURE OF
DIELECTRIC FLUID
PERIODICALLY DETERMINING
THE ENERGY NEEDED TO
COOL THE SERVERS
DETERMINE THE OPTIMAL
SECONDARY COOLING 158
METHOD TO MINIMIZE
ENERGY USAGE
DETERMINE PREFERABLE
SETTINGS FOR DIELECTRIC 160
FLUID PUMP FLOW ,
SECONDARY COOLING SYSTEM ,|
AND OPTIONALLY TANK FLUID
VELOCITY
EXECUTE OUTPUT CONTROL
SIGNALS TO PUMPS, VALVES,
AND FLUID VELOCITY
AUGMENTATION SYSTEMS
EXECUTE NOTIFICATION OF
FAILURE
Fig . 17B
US 10 , 123,463 B2
LIQUID SUBMERGED , HORIZONTAL the rack can be placed within a data center. Any of various
COMPUTER SERVER RACK AND SYSTEMS computing devices, such as , for example , network routers ,
AND METHOD OF COOLING SUCH A hard - drive arrays, data acquisition equipment and power
SERVER RACK supplies, are commonly mounted within a rack .
Data centers housing such servers and racks of servers
CROSS -REFERENCES TO RELATED typically distribute air among the servers using a centralized
APPLICATIONS fan (or blower ). As more fully described below , air within
the data center usually passes through a heat exchanger for
This application claims priority pursuant to 35 U . S . C . 119 cooling the air (e . g ., an evaporator of a vapor- compression
to the following U .S . provisional patent applications : 10 cycle refrigeration cooling system (or “ vapor- cycle ” refrig
Ser. No. 61/ 188 ,589 entitled LIQUID SUBMERGED , eration ), or a chilled water coil) before entering a server. In
HORIZONTAL COMPUTER SERVER RACK filed Aug . some data centers , the heat exchanger has been mounted to
11 , 2008 ; the rack to provide “ rack -level" cooling of air before the air
Ser. No . 61/ 163 , 443 entitled LIQUID SUBMERGED , enters a server . In other data centers, the air is cooled before
HORIZONTAL COMPUTER SERVER RACK filed Mar. 15 entering the data center.
25 , 2009; and In general, electronic components of higher performing
Ser. No. 61/ 165,470 entitled LIQUID SUBMERGED , servers dissipate correspondingly more power. However,
HORIZONTAL COMPUTER SERVER RACK filed Mar. power dissipation for each of the various hardware compo
31, 2009 . nents ( e . g ., chips, hard drives , cards ) within a server can be
20 constrained by the power being dissipated by adjacent
FIELD OF INVENTION heating generating components , the airflow speed and air
flow path through the server and the packaging of each
This application concerns cooling ofheat- generating elec - respective component, as well as a maximum allowable
tronics such as, for example, rack mounted servers in data operating temperature of the respective component and a
centers . 25 temperature of the cooling air entering the server as from a
data center housing the server. The temperature of an air
BACKGROUND stream entering the server from the data center , in turn , can
be influenced by the power dissipation and proximity of
In 2006 , data centers in the United States (U . S .) accounted adjacent servers , the airflow speed and the airflow path
for about 1.5 % (about $ 4.5 billion ) of the total electricity 30 through a region surrounding the server , as well as the
consumed in the U . S . This data center electricity consump- temperature of the air entering the data center ( or, con
tion is expected to double by 2011 . More than one -third of versely, the rate atwhich heat is being extracted from the air
data center electricity consumption is for cooling servers , within the data center ).
which could equate to more than about 1 % of all U .S . In general, a lower air temperature in a data center allows
electricity consumed by 2011. Electricity , personnel, and 35 each server component to dissipate a higher power , and thus
construction costs continue to increase and server hardware allows each server to dissipate more power and operate at a
costs are decreasing, making the overall cost of cooling a level of hardware performance . Consequently , data centers
large and growing part of the total cost of operating a data have traditionally used sophisticated air conditioning sys
center. tems ( e . g ., chillers, vapor -cycle refrigeration ) to cool the air
The term “ data center " ( also sometime referred to as a 40 ( e . g ., to about 65° F .) within the data center for achieving a
" server farm ” ) loosely refers to a physical location housing desired performance level. By some estimates , as much as
one or “ servers ." In some instances , a data center can simply one watt can be consumed to remove one watt of heat
comprise an unobtrusive corner in a small office . In other dissipated by an electronic component. Consequently , as
instances, a data center can comprise several large , ware energy costs and power dissipation continue to increase , the
house - sized buildings enclosing tens of thousands of square 45 total cost of cooling a data center has also increased .
feet and housing thousands of servers . The term “ server” In general, spacing heat- dissipating components from
generally refers to a computing device connected to a each other ( e . g ., reducing heat density ) makes cooling such
computing network and running software configured to components less difficult ( and less costly when considering ,
receive requests ( e. g ., a request to access or to store a file , for example , the cost of cooling an individual component in
a request to provide computing resources, a request to 50 a given environment) than placing the same components
connect to another client) from client computing devices , placed in close relation to each other (e.g., increasing heat
includes PDAs and cellular phones, also connected to the density ). Consequently , data centers have also compensated
computing network . Such servers may also include special for increased power dissipation ( corresponding to increased
ized computing devices called network routers , data acqui server performance ) by increasing the spacing between
sition equipment, movable disc drive arrays, and other 55 adjacent servers .
devices commonly associated with data centers . In addition , large - scale data centers have provided several
Typical commercially -available servers have been cooling stages for cooling heat dissipating components . For
designed for air cooling . Such servers usually comprise one example , a stream of coolant, e.g., water , can pass over an
or more printed circuit boards having a plurality of electri- evaporator of a vapor- compression refrigeration cycle cool
cally coupled devices mounted thereto . These printed circuit 60 ing system and be cooled to , for example , about 44° F .
boards are commonly housed in an enclosure having vents before being distributed through a data center for cooling air
that allow external air to flow into the enclosure , as well as within the data center.
out of the enclosure after being routed through the enclosure The power consumed by a chiller can be estimated using
for cooling purposes . In many instances , one or more fans information from standards ( e .g ., ARI 550 /590 - 98 ). For
are located within the enclosure to facilitate this airflow . 65 example , AR1550 /590 - 98 specifies that a new centrifugal
“ Racks” have been used to organize several servers. For compressor, an efficient and common compressor used in
example , several servers can be mounted within a rack , and high - capacity chillers, has a seasonal average Coefficient
US 10 , 123,463 B2
of-Performance (“ COP” ) from 5 .00 to 6 . 10 , depending on mary mode of heat dissipation . Refrigeration as a primary
the cooling capacity of the chiller. This COP does not mode of cooling, directly or indirectly , requires significant
include power consumed by an evaporative cooling tower, amounts of energy.
which can be used for cooling a condenser in the refrigera - Two - phase cooling systemshave been attempted , but due
tion cycle cooling system and generally has a COP of 70 , or 5 to technical complexity , they have not resulted in cost
better. The combined COP for a typical system is estimated effective products or sufficiently low operating costs to
to be about 4 .7 . justify investing in two-phase -cooling capital. Still other
single - and two- phase cooling systems bring the coolant
According to some estimates, some state -of-the-art data medium
centers are capable of cooling only about 150 Watts -per to an exterior of the computer, but reject heat to a
square -foot, as opposed to cooling the more than about 10 cooling medium (e .g ., air ) external to the computer and
1, 200 Watts -per- square - foot that could result from arranging within
ingly ,
the data center (e . g ., within a server room ). Accord
each method of server or computer cooling currently
servers to more fully utilize available volume (e.g., closely employed or previously attempted have been prohibitively
spacing servers and racks to more fully utilize floor- to expensive and/ or insufficient to meet increasing cooling
ceiling height and floor space ) within existing data centers . 15 demands of computing devices .
Such a low cooling capacity can significantly add to the cost Indirectly, many researchers have tried to reduce the
of building a data center, since data centers can cost as much power of individual components such as the power supply
as about $ 250 per-square - foot to construct. and CPU . Although chips capable of delivering desirable
As the air-cooling example implies , commercially avail performance levels while operating at a lower relative power
able methods of cooling have not kept pace with increasing 20 have been offered by chip manufacturers, such chips have ,
server and data -center performance needs , or the corre - to date , been expensive . Consequently , cooling approaches
sponding growth in heat density . As a consequence , adding to date have resulted in one or more of a high level of
new servers to existing data centers has become difficult and electricity consumption , a large capital investment and an
complex given the effort expended to facilitate additional increase in hardware expense .
power dissipation , such as by increasing an existing data 25 Therefore , there exists the need for an effective, efficient
center ' s air conditioning capacity . and low - cost cooling alternative for cooling electronic com
Various alternative approaches for cooling data centers ponents , such as, for example , rack -mounted servers.
and their servers , e . g ., using liquid cooling systems, have
met with limited success . For example, attempts to displace SUMMARY OF INVENTION
heat from a microprocessor (or other heat- generating semi- 30
conductor-fabricated electronic device component, collec systems Briefly , the present invention provides novel apparatus,
tively referred to herein as a “ chip ” ) for remotely cooling the devices,havingand methods for efficiently cooling computing
chip have been expensive and cumbersome. In these sys as, for example ,heat - generating electronic components, such
independently operable servers immersed in
tems, a heat exchanger or other cooling device j, .has has been
been 36
35 a dielectric liquid coolant in a tank .
placed in physical contact (or close physical relation using The system may include at least one tank defining an
a thermal- interface material) with the package containing interior volume and having a coolant inlet for re
interior volume and having a coolant inlet for receiving a
the chip . These liquid - cooled heat exchangers have typically dielectric liquid coolant within the interior volume and
defined internal flow channels for circulating a liquid inter - having a coolant outlet for allowing the dielectric liquid
nally of a heat exchanger body. However, component loca - 40 coolant to flow from the interior volume, the coolant inlet
tions within servers can vary from server to server. Accord and the coolant outlet being fluidly coupled to each other ;
ingly , these liquid -cooling systems have been designed for one or more mounting members positioned within the inte
particular component layouts and have been unable to rior volume and configured to mountably receive a plurality
achieve large-enough economies of scale to become com of independently operable servers; a dielectric liquid cool
mercially viable . 45 ant ; a heat exchanger fluidly coupled to the coolant outlet of
Research indicates that with state - of-the- art cooling, the at least one tank , the heat exchanger being distally
PUEs (as defined on page 10 hereinafter ) of 1.4 might be located from the tank ; a pump fluidly coupled to the heat
attainable by 2011 . However the costs to capitalize such exchanger and the interior volume of the at least one tank ,
cooling were not mentioned , and indicators suggest that the pump being configured for pumping the liquid coolant
saving electricity requires expensive equipment. 50 through a fluid circuit comprising a first circuit portion
Immersion cooling of electronic components has been extending from the coolant inlet of the tank to each server,
attempted in high -performance ( e.g ., computer gaming) a second circuit portion extending from each respective
applications, but has not enjoyed widespread commercial server to the coolant outlet, a third circuit portion extending
success . Previous attempts at immersion cooling has sub from the coolant outlet to the heat exchanger , and a fourth
merged some, and in some instances all , components 55 portion extending from the heat exchanger to the coolant
mounted to a printed circuit board in a dielectric fluid using inlet; a controller for monitoring the temperature of the
a hermetically sealed enclosure to contain the fluid . Such dielectric liquid coolant at at least one location within the
systems have been expensive , and offered by a limited fluid circuit and for adjusting the flow of the dielectric liquid
number of suppliers . Large scale data centers generally coolant through the fluid circuit in order that the dielectric
prefer to use " commoditized ” servers and tend to not rely on 60 liquid coolant is maintained at an elevated temperature as it
technologies with a limited number of suppliers . exits the second circuit portion of the fluid circuit; wherein
Control systemshave been used to increase cooling rates the at least one tank is configured for containing the dielec
for a plurality of computers in response to increased com - tric liquid coolant within the interior volume such that, when
putational demand . Even so , such control systems have the plurality of servers are mountably received therein , each
controlled cooling systems that dissipate heat into the data 65 server is submerged within the dielectric liquid coolant for
center building interior air (which in turns needs to be cooled sufficiently cooling each respective server while maintaining
by air conditioning ), or directly use refrigeration as a pri- the exiting heated liquid coolant at the elevated temperature
US 10 , 123,463 B2
to reduce the amount of energy consumed to sufficiently cool mately the elevated temperature to reduce the amount of
each of the plurality of servers . energy consumed to sufficiently cool each respective server.
Alternatively , the cooling system includes at least one The fixture or server rack apparatus includes at least one
tank defining an open interior volume; one or more mount- tank defining an open interior volume and having a coolant
ing members positioned within the open interior volume and 5 inlet for receiving a dielectric liquid coolantwithin the open
configured to mountably receive a plurality ofindependently interior volume and having a coolant outlet for allowing the
operable servers within the interior volume; a dielectric coolant to flow from the open interior volume, the coolant
liquid coolant circulating in a first fluid circuit through the inlet and the coolant outlet being fluidly coupled to each
plurality of servers; a secondary cooling system having a other ; and one or more mounting members positioned within
cooling fluid flowing in a second fluid circuit wherein the 10 the interior volume and configured to mountably receive a
secondary cooling system rejects heat from the cooling fluid ; plurality of servers in a vertical orientation within the
a coupler located within the at least one tank for thermally interior volume for minimizing the footprint of the server
coupling heated dielectric coolant from the portion of the relative to the ground and with the front of the server facing
first fluid circuit exiting the plurality of servers within the upward for easy installation and removal of each of the
tank to the cooling fluid in the second fluid circuit for 15 plurality of servers without removing or disturbing any other
rejecting heat from such heated dielectric coolant; a con - server ; wherein the at least one tank is configured for
troller for monitoring the temperature of the dielectric liquid containing a dielectric liquid coolant within the interior
coolant at at least one location within the first fluid circuit volume such that, when a plurality of servers are mountably
and for adjusting the flow of the cooling fluid through the received therein , each server being mountably received is
second fluid circuit in order that the heated dielectric liquid 20 submerged within the dielectric liquid coolant for suffi
coolant exiting the plurality of servers is maintained ciently cooling each respective server when the tank is
approximately at an elevated temperature wherein the sufficiently full of the liquid coolant.
elevated temperature is a temperature significantly higher server room fluidly connected to a first heat exchanger
than the typical comfortable room temperature for humans distally located from the server room contains the apparatus
and lower than themaximum permissible temperature of the 25 described above , including at least one tank defining an
most sensitive heat generating electronic component in the interior volume for containing a dielectric liquid coolant and
plurality of servers ; wherein the at least one tank is config - one or more mounting members positioned within the inte
ured for containing the dielectric liquid coolant within the rior volume and configured to mountably receive a plurality
interior volume such that, when the plurality of servers are of independently operable servers . The server room also
mountably received therein , at least a substantial portion of 30 contains a plurality of independently operable servers
each server is submerged within the dielectric liquid coolant wherein each of the plurality of servers is mountably
for sufficiently cooling each respective server when the tank received by the one or more mounting members such that
is sufficiently full of the liquid coolantmaintaining the liquid each of the respective servers is submerged in a volume of
coolant [exiting] the plurality of servers at approximately the dielectric liquid coolant for absorbing heat from each
elevated temperature to reduce the amount of energy con - 35 respective one of the plurality of servers . The server room
sumed to sufficiently cool each respective server. further contains at least one coupler for thermally coupling
Alternatively , the cooling system may include at least one the heated dielectric liquid coolant heated to the heat
tank defining an open interior volume; one or more mount exchanger for rejecting at least some of the heat absorbed by
ingmembers positioned within the open interior volume and the dielectric liquid coolant from each of the plurality of
configured to mountably receive a plurality of independently 40 servers . The heat exchanger may be associated with a
operable servers within the interior volume; a dielectric secondary cooling system . The coupler may include a fluid
liquid coolant circulating in a first fluid circuit through the coupler for fluidly coupling the dielectric liquid coolant to
plurality of servers ; a secondary cooling system having a the first heat exchanger . Alternatively , the coupler includes
cooling fluid flowing in a second fluid circuit wherein the a heat exchanger located internal to the tank and thermally
secondary cooling system rejects some of the heat from the 45 coupled to the dielectric liquid coolant heated by the servers
cooling fluid ; a coupler located within the at least one tank and a secondary fluid circuit with a second cooling fluid in
for thermally coupling heated dielectric coolant from the fluid connection between the distally located heat exchanger
portion of the first fluid circuit exiting the plurality of servers and the internally located heat exchanger wherein the dielec
within the tank to the cooling fluid in the second fluid circuit tric liquid coolant differs from the cooling fluid wherein the
for rejecting some of the heat from such heated dielectric 50 distally located heat exchanger is thermally coupled to the
coolant; a controller for monitoring the temperature of the cooling fluid flowing in the secondary fluid circuit such that
dielectric liquid coolant at at least one location within the the distally located heat exchanger rejects heat from the
first fluid circuit and for adjusting the flow of the cooling cooling fluid which the cooling fluid has absorbed from the
fluid through the second fluid circuit in order that the heated heated dielectric liquid coolant at the coupler.
dielectric liquid coolant exiting the plurality of servers is 55 A method of cooling a plurality of independently operable
maintained approximately at an elevated temperature servers includes flowing a dielectric liquid coolant in a fluid
wherein the elevated temperature is a temperature signifi- circuit through the plurality of servers immersed within the
cantly higher than the typical comfortable room temperature dielectric liquid coolant for absorbing at least a portion of
for humans and lower than the maximum permissible tem - any heat being dissipated by each of the respective servers ;
perature of the most sensitive heat generating electronic 60 monitoring the temperature of the liquid coolant at at least
component in the plurality of servers ; wherein the at least one location within the fluid circuit; determining the opti
one tank is configured for containing the dielectric liquid mum elevated temperature of the heated dielectric liquid
coolant within the interior volume such that, when the coolant as it exits the plurality of servers such that the liquid
plurality of servers are mountably received therein , each coolant sufficiently cools the plurality of servers while
server is submerged within the dielectric liquid coolant for 65 reducing the amount of energy consumed to sufficiently cool
sufficiently cooling each respective server and maintaining each respective server, wherein the elevated temperature is
the liquid coolant exiting the plurality of servers at approxi- a temperature significantly higher than the typical comfort
US 10 , 123 ,463 B2
able room temperature for humans and lower than the FIG . 14 is another end elevation view of the immersion
maximum permissible temperature of the most sensitive heat cooled server racks of FIG . 11 showing the flow of the liquid
generating electronic component in the plurality of servers ; coolant;
periodically determining by a controller the amount of FIG . 15 is a schematic illustration of a system for cooling
energy needed to reject the absorbed heat for cooling the 5 a plurality of immersion -cooled server racks of the type
plurality of servers; thermally coupling the dielectric liquid shown in , for example, FIG . 3 and installed in a server room .
coolant heated by the plurality of servers to a heat exchanger FIG . 16 illustrates an exemplary method of cooling one or
distally located from the tank ; and rejecting at least a portion more independently operable servers immersed in a tank of
of the heat absorbed by the liquid coolant. In response to the liquid coolant employing the systems of FIG . 1A or 1B ;
periodic determination of the amount of energy needed to 10 FIG . 17A illustrates the physical steps in the method of
reject the heat absorbed by the dielectric liquid coolant from cooling one or more independently operable servers
the servers by a controller, the method may also include the immersed in a tank of liquid coolant employing the system
step of periodically adjusting the amount of heat rejected of FIG . 1A ; and
through the heat exchanger such that the dielectric liquid 1 FIG . 17B illustrates the computer controller -based steps
coolant exiting the plurality of servers at the elevated in the method of cooling one or more independently oper
temperature sufficiently cools the plurality of servers while able servers immersed in a tank of liquid coolant employing
reducing the amount of energy consumed to sufficiently cool the system of FIG . 1A .
each respective server .
20 DETAILED DESCRIPTION
BRIEF DESCRIPTION OF THE DRAWINGS
The following describes apparatus , systems, and methods
For a more complete understanding of the present inven - for efficiently cooling computing devices having heat - gen
tion (s ), and the advantages thereof, reference is now made to erating electronic components, such as, for example , inde
the following descriptions taken in conjunction with the 25 pendently operable servers at least partially immersed in a
accompanying drawings , in which : dielectric liquid coolant in a tank . The principles of the
FIG . 1A illustrates one embodiment of an exemplary invention (s ) embodied therein and their advantages are best
system for efficiently cooling a plurality of independently understood by referring to FIGS. 1- 17 .
operable servers ; As used herein , the term “ server” generally refers to a
FIG . 1B illustrates an alternative embodiment of an 30 computing device connected to a computing network and
exemplary system for efficiently cooling a plurality of running software configured to receive requests ( e .g ., a
independently operable servers ; request to access or to store a file , a request to provide
FIG . 2 illustrates the system of FIG . IA in more detail ; computing resources, a request to connect to another client)
from client computing devices, includes PDAs and cellular
FIG . 3 illustrates a perspective view of an exemplary 35 phones
immersion -cooled rack having a plurality of independently servers , may also connected to the computing network . Such
also include specialized computing devices
operable servers mounted therein .
FIG . 4 illustrates a top plan view of the immersion - cooled called blade servers,network routers , data acquisition equip
ment, movable disc drive arrays, and other devices com
rack shown in FIG . 3 . monly associated with data centers.
FIG . 5 illustrates an end elevation view of the immersion - 40 As used herein . “ independently operable ” means capable
cooled rack shown in FIG . 3 . of usefully functioning without regard to an operational
FIG . 6 illustrates a side elevation view of the immersion status of an adjacent component. As used herein , an “ inde
cooled rack shown in FIG . 3 . pendently operable server” means a server that is capable of
FIG . 7 illustrates an end elevation view of an alternative usefully functioning (e . g ., powered or unpowered , con
immersion -cooled rack having a plurality of independently 45 nected to a network or disconnected from the network ,
operable servers installed therein . installed in a rack or removed from a rack , and generally
FIG . 8 illustrates a top plan view of the immersion -cooled used for the purposes for which servers are generally used )
rack shown in FIG . 7 . without regard to an operational status of an adjacent server
FIG . 9 illustrates an end elevation view of another alter (e . g ., powered or unpowered , connected to the network or
native immersion - cooled rack having a plurality of indepen - 50 disconnected from the network , installed in the rack or
dently operable servers mounted therein . removed from the rack , and whether usable for the purposes
FIG . 10 illustrates an end elevation view of yet another for which servers are generally used ). Operation of inde
alternative immersion -cooled rack having a plurality of pendently operable servers can be influenced (e.g ., heated )
independently operable servers mounted therein . by one or more adjacent servers , but as used herein , an
FIG . 11 illustrates a perspective view of side -by -side 55 independently operable server generally functions regard
immersion - cooled racks having a plurality of independently less of whether an adjacent server operates or is operable .
operable servers mounted therein with the electrical con - As used herein , the term “ liquid coolant” may be any
nections to the servers shown. sufficiently non - conductive liquid such that electrical com
FIG . 12A is perspective view of one version of a con - ponents (e . g ., a motherboard , a memory board , and other
ventional rack -mountable server that may be installed in the 60 electrical and /or electronic components designed for use in
exemplary immersion -cooled server racks depicted in FIGS . air ) continue to reliably function while submerged without
3 through 11; significantmodification . A suitable liquid coolant is a dielec
FIG . 12B is an illustration of a hard drive of the conven tric liquid coolant, including without limitation vegetable
tional rack -mountable server of FIG . 12A with a liquid oil,mineral oil ( otherwise known as transformer oil), or any
proof enclosure to be inserted around it ; 65 liquid coolant have similar features ( e . g ., a non - flammable ,
FIG . 13 is an end elevation view of the immersion - cooled non -toxic liquid with dielectric strength better than or nearly
server racks of FIG . 11; as comparable as air.
US 10 , 123 , 463 B2
10
As used herein , “ fluid ” means either a liquid or a gas, and ference between heat generating (or dissipating ) compo
“ cooling fluid ” means a gas or liquid coolant typically used nents and a cooling medium (also referred to herein as
for heat-rejection or cooling purposes. As used herein , a " coolant” or “ liquid coolant" ) used to cool the components
liquid coolant is a subset of the universe of cooling fluids, by maintaining a coolant temperature (e. g ., an average bulk
but a cooling fluid may be a dielectric or non -dielectric 5 fluid temperature ) at an acceptably elevated temperature
liquid or gas, such as , for example, a conventional air compared to conventional cooling technologies. Such an
conditioning refrigerant. elevated coolant temperature can reduce the power con
PUE means “ power usage effectiveness ” , which is a ratio sumed for cooling purposes ( e. g., heat can be more readily
ofthe total power used by a data center divided by the power rejected from a “ high -temperature ” coolant to the environ
used by the server, and is a measure of energy efficiency . 10 ment than from a “ low -temperature” coolant).
COP means the " coefficient of performance” , a ratio of FIG . 1A illustrates one embodiment of cooling system
heat removed to work used . For instance , a COP of 10 would 100 for cooling a rack of independently operable servers .
mean that 10 Watts of heat are removed using 1 Watt of The system 100 includes a tub or tank 110 containing a
work . dielectric liquid coolant into which a plurality of servers 120
VCC means “ vapor compression cycle ” , the thermal 15 may be immersed . Mounting members or rails to be
process most commonly used for air conditioning. described hereinafter are positioned within the interior vol
Poor overall efficiency of heretofore commercially avail- ume of the tank 110 and are configured to receive and mount
able cooling technologies contributes to overall costs of the plurality of servers 120 as a rack of servers into the tank
cooling servers used by data centers . As disclosed herein , 110 . Such a tank 110 may have an opening for access to each
applicants have discovered that the irreversibilities contrib - 20 of the servers mounted in the rack . At least a portion of each
uting to this poor overall efficiency can be reduced , reducing server 120 is submerged within the dielectric liquid coolant
the overall cost of cooling servers ( as well as the corre for sufficiently cooling each respective server when the tank
sponding cost of operating data centers ). 110 is sufficiently full of the liquid coolant. Preferably , each
As between two bodies ( or fluids ) at different tempera - of the servers during operation is completely submerged
tures , heat flows from the higher- temperature body to the 25 within the dielectric liquid coolant.
lower-temperature body. For a given amount of transferred The liquid coolant heated by the servers 120 in the server
heat, such heat transfer is less irreversible (e . g ., the associ- rack is then fluidly coupled through suitable piping or lines
ated energy retains more " usefulness,” or is of a “ higher to a pump 130, which pumps the heated liquid coolant
quality ” ) when both temperatures are higher as compared to through suitable piping or lines to a remotely or distally
a heat transfer process occurring at lower temperatures . 30 located heat exchanger 140 associated with a heat- rejection
Methods , systems, and apparatus are disclosed for efficiently or cooling apparatus 150. The distally heat exchanger 140
cooling heat- generating electronic components , as by trans- rejects the heat from the incoming heated liquid coolant and
ferring heat from the components at a first temperature ( e . g ., fluidly couples the cooled liquid coolant through a return
about 158° F . in some instances ) to a liquid coolant at a fluid line or piping 170 back into the tank 110 . Thus , at least
“ high ” temperature (e . g ., a dielectric liquid coolant such as, 35 a portion of the liquid coolant completes a fluid circuit
for example, mineral oil at a temperature of, for example , through the servers 120 in the tank 110 , pump 130 , heat
about 105° F .). Such heat transfer from the heat- generating exchanger 140 , and back into the tank 110 . The heat rejected
components at the first temperature to a coolant at a " high ” from the heated liquid coolant through the heat exchanger
temperature can be less irreversible than transferring the 140 may then be selectively used by alternative heat rejec
same quantity of heat from the components at the first 40 tion or cooling apparatus 150 to be described hereinafter to
temperature to a coolant at a “ low ” temperature ( e .g ., air at dissipate , recover , or beneficially use the rejected heat
a temperature of, for example, 65° F .). depending on the different environmental conditions and /or
The methods, systems, and apparatus disclosed herein server operating conditions to which the system is subject.
take advantage of this thermodynamic principle to improve The system 100 includes a computer controller 180 of
the overall efficiency of cooling electronic components , as 45 conventional design with suitable novel applications soft
can be applied to , for example , independently operable ware for implementing the methods of the present invention .
servers of the type commonly used in a data center. Such The controller 180 may receive monitor signals of various
improved cooling efficiency can reduce the overall cost of operational parameters from various components of the
operating a data center by reducing electricity consumed for cooling system 100 and the environment and may generate
cooling purposes . 50 control signals to control various components of the cooling
In some disclosed embodiments, the reduced temperature system to maintain the heated liquid coolant exiting the
differences ( resulting in lower irreversibility ) allows for heat servers in the tank at a specific elevated temperature in order
to be recaptured . In other embodiments , the reduced tem - to sufficiently cool each of the servers while reducing the
perature differences reduces (or altogether removes ) the total amount of energy needed to cool the servers . Particu
need for refrigeration . In all of the disclosed embodiments, 55 larly , the controller 180 monitors the temperature of the
the corresponding cooling cycle efficiency of the cooling liquid coolant at at least one location within the fluid circuit ,
system increases as compared to conventional, commer for example where the heated liquid circuit exits the plural
cially available cooling cycles. ity of servers. The controller 180 may also monitor the
temperature of the heat- generating electronic components in
Overview 60 the servers in the server racks by electrically connecting the
controller 180 to the diagnostic output signals generated by
FIG . 1A and FIG . 1B depict alternative exemplary sys- conventional rack -mountable servers. The controller may
tems 100 and 200 , respectively, for cooling one or more also monitor the flow of the dielectric liquid coolant. Based
independently operable servers containing heat-generating upon such information , the controller 180 may output sig
electronic components , such as can be arranged in one or 65 nals to the pump 130 and heat rejection or cooling apparatus
more server racks, for example , in a data center. Some 150 to adjust the flow of the liquid coolant through the fluid
disclosed systems and methods reduce the temperature dif- circuit and the amount of the heat being rejected by the heat
US 10 , 123 , 463 B2
12
rejection or cooling apparatus 150 for sufficiently cooling such information , the controller 180 may output signals to
each respective server while maintaining the heated liquid the heat rejection or cooling apparatus 250 to adjust the flow
coolant exiting the servers at the elevated temperature to of the cooling liquid through the external fluid circuit and the
reduce the amount of energy consumed to sufficiently cool amount of the heat being rejected by the heat rejection or
each of the servers in the server rack . 5 cooling apparatus 250 for sufficiently cooling each respec
FIG . 1B illustrates one embodiment of an alternative tive server while maintaining the heated liquid coolant
cooling system 200 for cooling a rack of independently exiting the servers at the elevated temperature to reduce the
operable servers. The system 200 includes a tub or tank 210 amount of energy consumed to sufficiently cool each of the
containing a liquid dielectric coolant into which a plurality
of servers 120 (not shown ) can be immersed . Mounting 10 servers . Preferably, the elevated temperature is a tempera
ture significantly higher than the typical comfortable room
members to be described hereinafter are positioned within temperature for humans and lower than the maximum per
the interior volume of the tank 210 and are configured to
receive and mount the plurality of servers 120 as a rack of missible temperature of the most sensitive heat generating
electronic component in the servers.
servers into the tank 210 . Such a tank 210 may have an open
ton for access to each of the servers mounted in the rack At 15 As previously described , a computer controller is used
least a portion of each server 120 is submerged within the control different components of the cooling system to main
dielectric liquid coolant for sufficiently cooling each respec - tain the exiting dielectric liquid coolant temperature at an
tive server when the tank 210 is sufficiently full of the liquid acceptable elevated temperature . By maintaining the exist
coolant. Preferably , each of the servers during operation is ing coolant at an elevated level, the cooling system may be
completely submerged within the dielectric liquid coolant. 20 used with a number of different techniques for using or
Unlike the cooling system 100, heated dielectric liquid dissipating the heat ( e.g ., heat recapture, low power heat
coolant does not flow outside the tank 210 . Instead , the fluid dissipation , or refrigeration ).
circuit 260 of the flowing dielectric liquid coolant is com - In some embodiments, an average bulk fluid temperature
pletely internal to the tank 210 . A thermal coupling device of the coolant can be maintained at a temperature of about,
280 , such as a heat exchanger , is mounted within the tank 25 for example , 105° F., which is significantly higher than a
210 within the fluid circuit through the servers so that at least typical room temperature , as well as the maximum average
a portion of the heated dielectric liquid coolant flow exiting outdoor temperature by month in the U . S . (e. g., about 75° F .
the servers flows through the thermal coupling device 280 . during summer months ). At a temperature of about 105° F .,
Cooled dielectric liquid coolant exits the coupling device heat can be rejected to the environment ( e . g ., the atmosphere
280 and at least a portion of the cooled dielectric coolant 30 nearby cooling sources such as rivers ) with little power
circulates in the internal fluid circuit 260 back through the or
consumed , or recaptured as by, for example , heating the
servers.
The system 200 includes a secondary heat rejection or same ing
or an adjacent building ' s hot-water supply or provid
indoor heating in cold climates.
cooling apparatus 250 having a cooling fluid , such as a gas
or liquid flowing in piping or lines , forming a second fluid 35 By maintaining a coolant temperature in excess of natu
circuit 290 wherein the secondary cooling apparatus 250 peraturerally occurring temperatures, irreversibilities and/ or tem
includes an associated remotely or distally located heat differences present in a server cooling system may
exchanger (not shown ) that rejects heat from the cooling be reduced . A reduction in irreversibilities in a thermody
fluid in the second fluid circuit through the distally remote namic cycle tends to increase the cycle 's efficiency , and may
heat exchanger . 40 reduce the overall power consumed for cooling the servers .
In a conventional cooling system , about one -half watt is
The heat rejected from the heated cooling fluid in the consumed
second fluid circuit through the heat exchanger associated by the cooling system for each watt of heat
with the secondary cooling apparatus 250 may then be (generated in a component. For example , a cooling medium
selectively dissipated , recovered , or beneficially used toe . gbe., aircooled
) can be cooled to about 65° F . and the components
can operate at a temperature of about, for
depending on the different environmental conditions and /or 45 example
" , 158° F . This large difference in temperature results
server operating conditions to which the system is subject. in correspondingly large inefficiencies and power consump
The system 200 includes a computer controller 270 with
suitable novel applications software for implementing the tion . In addition , the " quality " of the rejected heat is low ,
methods of the present invention . The controller 180 may making the heat absorbed by the cooling medium difficult to
recapture after being dissipated by the component(s). How
receive monitor signals of various operational parameters 50 ever , with a cooling medium such as air, such a large
from various components of the cooling system 200 and the temperature
environment and may generate control signals to control difference may be necessary in conventional
various components of the cooling system to maintain the systems in order to achieve desired rates of heat transfer.
heated liquid coolant exiting the servers in the tank 210 at a For example , one - dimensional heat transfer, Qi- D , can be
specific elevated temperature in order to sufficiently cool 55 modeled as the quotient of a temperature difference ,
each of the plurality of servers while reducing the total AT, divided by a thermal resistance, Rih
amount of energy needed to cool the servers . Particularly,
the controller 270 monitors the temperature of the liquid
coolant at at least one location within the internal fluid
circuit, for example , where the heated liquid circuit exits the 60 (ie., Q1-4 = RM)
servers immersed in the tank. The controller 270 may also
monitor the temperature of the heat- generating electronic Accordingly, for a given heat dissipation from a compo
components in the servers in the server racks by electrically nent, a temperature difference between the component and a
connecting the controller to the diagnostic output signals stream of liquid coolantneeds to be larger for higher thermal
generated by conventional rack -mountable servers , The con - 65 resistance than for a lower thermal resistance . Typically, a
troller may also monitor the flow and temperature of the flow of gas ( e . g ., air ) has a higher thermal resistance value
cooling fluid in the external fluid circuit 290 . Based upon than a flow of liquid (e .g ., a dielectric liquid coolant).
US 10 , 123,463 B2
13 14
Accordingly, a gas cooling fluid typically requires a larger upon such information , the computer controller may output
temperature difference than a liquid coolant. signals to the pump 330 and heat rejection or cooling
apparatus 350 to adjust the flow of the liquid coolant through
Illustrative Embodiments of the System and the fluid circuit and the amount of the heat being rejected by
Apparatus 5 the heat rejection or cooling apparatus 350 for sufficiently
cooling each respective server when the tank 310 is suffi
In FIG . 2 , the cooling system 300 illustrates one embodi ciently full of the liquid coolant while maintaining the
ment of the cooling system 100 of FIG . 1A in more detail.
The system 300 includes a tub or tank 310 containing a heated temperature
liquid coolant exiting the servers at the elevated
to reduce the amount of energy consumed to
liquid coolant into which a plurality of servers 120 can be 10 sufficiently cool each of the plurality of servers . In addition ,
immersed . Mounting members to be described hereinafter the controller 380 also may operate an optimization program
are positioned within the interior volume of the tank 310 and within the applications software as discussed hereinafter to
are configured to receive and mount the plurality of servers determine which of the alternative heat rejection apparatus
as a rack of servers into the tank 310 . Such a tank 310 may 350 connected to the fluid valve 390 provides the most
have an opening for access to each of the servers mounted 15 efficient means of rejecting the heat from the heated liquid
in the rack . At least a portion of each server 120 is sub
merged within the liquid coolant for sufficiently cooling ditionscoolant given the environmental and server operating con
each respective server when the tank 310 is sufficiently full . It should be noted , however, that the cooling system
of the liquid coolant. Preferably, each of the servers during 300 does not necessarily require different methods of heat
operation is completely submerged within the liquid coolant. 20 todissipation . In some instances it may be more cost effective
The liquid coolantheated by the servers 120 in the server In FIGS . 3 thruone6 ,.a suitable fixture or rack apparatus 400 for
only have
rack is then fluidly coupled through suitable piping or lines
to a pump 330 , which pumps the heated liquid coolant immersing liquid coolant
a rack of independently operable servers in a
422 is depicted . The apparatus 400 includes a
through suitable piping or lines through a filter 360 to one or
more fluid valves 390 . The fluid valve 390 may be remotely 25 tub or tank 410 and mounting members for mounting the
servers , as will be described in more detail hereinafter. The
controlled to connect the heated liquid coolant being tank 410 may be fabricated of steel, a sufficiently strong
pumped through the collection piping from the tank 310 to plastic that is compatible with the dielectric liquid coolant
a controller -selected one of alternative remotely or distally
located heat exchangers associated with alternative heat used as a cooling medium , or other suitable material. The
rejection or cooling apparatus 350, such as an outside air 30 tank 410 may face upward with an open top 430 to form an
open interior volume and may be shaped to have a length L ,
radiator 352 permitting cooling with outside ambient atmo width W , and height H with the minimum footprint to insert
spheric air , a refrigeration system 354 , a heat recovery
system 356 , or an evaporative cooler 358. The distally multiple used to
servers 120 . Suitable mounting members may be
mount the servers in the tank to form the server rack
located heat exchanger associated with a selected one of the
alternative heat rejection or cooling apparatus 350 then 35 470 within the tank . The tank 410 may be shaped and the L ,
W , and H dimensions sized such thatmultiple standard - sized
rejects the heat from the incoming heated liquid coolant and servers
fluidly couples the cooled liquid coolant through a return shown , intypically measured in units of “ U ” or 1 .75 inches (as
fluid line or piping 370 back into the tank 310 . Thus, at least modification . ), can be supported without significant
FIG . 4
a portion of the liquid coolant completes a fluid circuit
through the servers 120 in the tank 310 , pump 330 , a heat 40 fromThea piping
tank is fabricated to have an inlet pipe or line 440
system connected to a heat exchanger for the
exchanger associated with a heat-rejection apparatus 350, flow of lower temperature or cooled liquid coolant into the
and back through piping 370 into the tank 310 . The heat
rejected from the heated liquid coolant through the heat tank 410 and an outlet pipe or line 450 connected to
exchanger may then be used by the selected one of alterna collection piping for the flowing or pumping of heated
coolant out of the tank to the external heat exchanger
tive heat rejection or cooling apparatus 350 to dissipate , 45 °Cassociated with one or more of the heat- rejection or cooling
recover , or beneficially use the rejected heat depending on systems described in connection with FIGS. 1A , 1B , and 2 .
the different environmental conditions and /or server operat The server rack itself may have a number of different
ing conditions to which the cooling system 300 is subject .
The cooling system 300 includes a computer controller 50 implementations
configured to
. Preferably , the mounting members are
mountably receive the plurality of servers in a
380 with suitable applications software which may receive 50 vertical
con orientation , thereby minimizing the footprint of the
monitor signals of various operational parameters from servers relative to the ground , and with the “ front” ! panel
various components of the system 300 and the environment facing upward for easy installation and removal of a server
and may generate control signals to control various compo
without the need to remove or disturb any other server
nents of the system 300 to maintain the heated liquid coolant within
exiting the servers in the tank at a specific elevated tem - 55 1 Upwardstheis tank 410 .
defined as one of the two smallest sides of a rectangular
perature in order to sufficiently cool each of the plurality of computer. The “ back ” is generally referred to as the side with wires inserted ,
servers while reducing the total amount of energy needed to such as power, communications , etc.
cool the servers . Similar to previous embodiments , the The mounting members may be also configured to mount
controller 380 monitors the temperature of the liquid coolant the servers such that the top level 460 of the liquid coolant
at at least one location within the fluid circuit, for example 60 completely submerges the top level 472 of the server rack
where the heated liquid coolant exits the plurality of servers.470 formed by the multiple servers 120. As a consequence ,
The controller may also monitor the temperature of the a volume of liquid coolant collects in a common manifold
heat- generating electronic components in the servers 120 in area above the server rack 470 to improve the circulation of
the server racks by electrically connecting the controller 380 the liquid coolant through the plurality of servers , thereby
to the diagnostic output signals generated by conventional 65 enhancing the cooling of each respective server. The mount
servers . The controller 380 may also monitor the flow of the ing members may also be configured to mount the servers in
liquid coolant through the tank and / or fluid circuit. Based the server rack 470 above the bottom of the tank to create a
US 10 , 123,463 B2
15 16
volume of liquid coolant between each respective server and configuration , the fluid flow 560 of the liquid coolant
the bottom of the tank such that the flow of the dielectric entering the tank through the inlet piping is initially through
liquid coolant through the servers is improved . Preferably, the volume 562 of liquid coolant formed by the longer side
the mounting members are configured to mount the servers of the tank containing the inlet piping 540 and the side 572
closely adjacent to one another in the server rack to restrict 5 of the server rack 570 of servers 120 and then through the
the flow of the dielectric liquid coolant between the verti- side 572 of the server rack through the servers 120 and out
cally - oriented servers , such that the flow of the dielectric the opposite side 574 of the server rack into a volume 576
liquid coolant through the servers is enhanced . of liquid coolant formed by the side 574 of the server rack
A pump, such as pump 330 in FIG . 2 , may pump liquid and the longer side of the tank containing the outlet piping
coolant from the external heat exchanger through the piping 10 550 .
system into the tank 410 to maintain coolant fluid circulation FIG . 9 depicts an end elevation view of yet another
within the tank . The liquid coolant may flow through each illustrative embodiment of a suitable fixture or server rack
installed server and exit at the server side positioned oppo - apparatus 600 for use in connection with a combination of
site the inlet to the tank . In FIGS. 3 thru 6 , the inlet piping system 100 of FIG . 1A and system 200 of FIG . 1B . In such
440 is located at one end of the rectangular tank 410 near the 15 a combination , there are two alternative modes of operating
bottom of the tank ; whereas the outlet piping 450 is located the cooling system for cooling the dielectric liquid coolant
nearer the top of the tank . This configuration permits the wherein the controller may switch the mode of operation
liquid coolant heated by the heat generating components in depending on the environmental conditions. The tank 610 is
the servers to naturally rise through the servers and exit shaped and sized like the embodiment shown in FIGS. 3 -6
through the top or “ front panel” of the servers. 20 except as noted herein below . The tank 610 also may have
The servers may be configured to minimize mixing of the an open top to form an open interior volume into which the
incoming liquid coolant with outgoing liquid coolant. Each servers may be mounted in a vertical orientation with the
tank may be shaped (or have a member installed ) to reduce front panel facing upward toward the open top of the tank .
the flow of coolant around the installed server ( e. g ., to The inlet piping 640 is located nearer one end of one of the
reduce by -pass flow ), thereby improving coolant flow over 25 longer sides of the rectangular tank than the middle and is
each heat generating component and /or respective heat sink located nearer the bottom of the tank than the middle . The
in each of the multiple servers. output piping 650 is located nearer the opposite end of the
Alternatively , the location of the piping 440 and 450 may same longer side of the rectangular tank nearer the top of the
be reversed such that the heated liquid coolantmay exit from tank . In the first mode of operation utilizing a mode of
the installed servers through its " rear" panel) into the outlet 30 operation comparable to that of FIG . IA , the fluid flow 660
into the collection piping system . The collection piping of the liquid coolant entering the tank through the inlet
transports the heated liquid coolantto the heat exchanger for piping 640 is initially through the space 662 formed by the
rejecting at least some of the heat absorbed from the bottom of the side of the tank containing the inlet piping 640
installed servers . and the bottom 672 of the server rack 670 of servers 120 and
In another alternative rack design ( not shown in the 35 then through the bottom 672 of the server rack through the
drawings ), the tank 410 is divided into a plurality of bins servers 120 and out the front panel side 674 of the server
with each bin being sized to receive one corresponding rack into a space 676 formed by the top 674 of the server
server with the “ front panel” facing upward . The external rack and the top surface 622 of the liquid coolant and nearer
pump pumps coolant from the external heat exchanger the outlet piping 650 . To permit the second mode of opera
through the piping system into each bin to maintain a coolant 40 tion similar to FIG . 1B , a second heat exchanger 680
fluid circulation within the tank and each respective bin . The associated with an additional secondary cooling apparatus is
liquid coolant may flow through each installed server and mounted within the tank 610 and a second inlet piping 682
exit at a side positioned opposite the inlet to the tank and /or and a second output piping 684 are inserted through the wall
inlet to the bin . In addition , each bin may be configured to of the tank 610 and fluidly coupled to the heat exchanger to
minimize mixing of the incoming liquid coolant with out- 45 permit the flow of a separate second cooling fluid through
going liquid coolant. Each bin may be shaped (or have a the input piping 682 , second heat exchanger 680 , and outlet
member installed ) to reduce the flow of coolant around the piping 684 back to the second secondary cooling apparatus.
installed server ( e. g ., to reduce by -pass flow ), improving In the second mode of operation , the pump associated
coolant flow over each heat- generating component and/ or with the first mode of operation is deactivated by the
respective heat sink in each of the servers . 50 controller such that the fluid circuit flow of the dielectric
FIGS. 7 and 8 depict another illustrative embodiment of liquid coolant to the external heat exchanger of the first
a suitable fixture or server rack apparatus 500 for immersing secondary cooling apparatus is deactivated . Next the internal
a rack of independently operable servers in a liquid coolant heat exchanger 680 associated with the second alternative
522 wherein the surface of the liquid coolant 524 is above secondary cooling apparatus is activated by the controller. In
the top of the server rack . FIG . 7 shows an end elevation 55 this mode the fluid flow of the dielectric fluid within the tank
view of the apparatus 500 , which includes a tub or tank 510 is reconfigured such that the heated dielectric liquid coolant
mounted on a mount 515 into which the servers 120 are fluid flow 660 flowing out of the servers 120 does not flow
submerged . The tank 510 may have an open top to form an out of the outlet piping 650 . Instead , at least a portion of the
open interior volume into which the servers may bemounted liquid coolant fluid flow 660 is through the heat exchanger
in a vertical orientation with the front panel facing upward 60 680 to the bottom of the tank 610 and then back through the
toward the open top of the tank . The tank 510 is shaped and servers 120 . The heat rejected from the heat exchanger 680
sized like the embodiment shown in FIGS . 3 - 6 except as is thermally coupled to the second cooling fluid of the
otherwise noted herein below . The inlet piping 540 is located second secondary cooling system for dissipation or recov
near one end of one of the longer sides of the rectangular ery .
tank near the bottom of the tank . The output piping 550 is 65 FIG . 10 depicts an end elevation view of yet another
located at the opposite end of the opposing longer side of the illustrative embodiment of a suitable fixture or server rack
rectangular tank also near the bottom of the tank . In this apparatus 700 for use in connection with a combination of
US 10 , 123,463 B2
17 18
system 100 of FIG . 1A and system 200 of FIG . 1B . In such FIGS. 11 , 13 and 14 depict another illustrative embodi
a combination , there are two different modes of operating ment of a suitable fixture or rack apparatus 800 for immers
the cooling system for cooling the dielectric liquid coolant. ing side - by - side immersion - cooled server racks of standard
The tank 710 is shaped and sized like the embodiment commercially available versions of independently operable
shown in FIGS. 3 -6 except as noted herein below . The tank 5 servers , such as those depicted in FIG . 12A for example , in
710 also may have an open top to form an open interior
volume into which the servers 120 may be mounted in a top servers shown . The tank 810 may face upward with an open
horizontal orientation with the front panel facing toward the to have 812 to form an open interior volume and may be shaped
shorter side of the rectangular tank in which the inlet piping a length L , width W , and height H with the minimum
740 is located . The inlet piping 740 is located nearer one end servers 820to .insert
10 footprint two rows or racks 830 and 832 ofmultiple
The tank 810 may be shaped and the dimensions
of one of the shorter sides of the rectangular tank than the
sized such thatmultiple standard -sized servers 820, typically
middle and is located nearer the bottom of the tank than the measured
middle . The output piping 750 is located nearer the opposite in units of “ U ” or 1.75 inches (as shown in FIG .
12A ), can be supported in two racks without significant
end of the same shorter side of the rectangular tank nearer 15 modification . Suitable mounting members may be used to
the top of the tank . In the first mode of operation utilizing a mount the servers in the tank to configure the server rack 830
mode of operation comparable to that of FIG . 1A , the fluid and 832 within the tank . Specifically , the mounting members
flow 760 of the liquid coolant entering the tank through the (not shown ) may be fixedly attached along the length L of
inlet piping 640 is initially through the space 762 formed by each longer side ofthe tank 810 and in themiddle of the tank
a longer side of the tank and the lower side 772 of the server 20 810 between the two shorter ends of the tank to support the
rack 770 of servers 120 and then through the bottom 772 of rack ears 836 of a standard rack -mountable server 820
the server rack through the servers 120 and out the front shown in FIG . 12A .
panel 774 of the server rack into a space 776 formed by the The tank may be fabricated to have an inlet pipe or line
front 774 of the server rack and the shorter side of the tank from a piping system connected to a heat exchanger for the
nearer the outlet piping 750. To permit the second mode of 25 flow of lower temperature or cooled liquid coolant into the
operation similar to FIG . 1B , a second heat exchanger 780 tank 810 and an outlet pipe or line connected to collection
associated with an additional secondary cooling apparatus is piping for the flowing or pumping of heated coolant out of
mounted within the tank 710 and a second inlet piping 782 the tank to the distally located heat exchanger as shown in
and a second output piping 784 are inserted through the wall FIG . 3 . After the two racks of multiple servers are mounted
of the tank 710 and fluidly coupled to the heat exchanger to 30 inside the tank 810 , the level 824 of the liquid coolant 822
permit the flow of a separate second cooling fluid through may be carefully controlled to adjust the amount of flow of
the input piping 782, second heat exchanger 780 , and outlet the liquid coolant through the multiple servers and to adjust
piping 784 back to the second secondary cooling apparatus. the amount of heat removal from the heat generating elec
In the second mode of operation , the pump associated tronic components in the servers.
with the first mode of operation is deactivated by the 35 Orienting the servers in a vertical orientation with the
controller such that the fluid circuit flow of the dielectric front panel facing upward may also be advantageous due to
liquid coolant to the external heat exchanger of the first the typical movable hard drive installation in a standard
secondary cooling apparatus is deactivated . Next the internal commercially available server. When a standard server such
heat exchanger 780 associated with the second alternative as shown in FIG . 12A , is oriented vertically the hard drive
secondary cooling apparatus is activated by the controller. In 40 890 of such a server , as shown in FIG . 12B , is oriented
this mode the fluid flow of the dielectric fluid within the tank vertically with the cables connecting at the bottom of the
is reconfigured such that the heated dielectric liquid coolant drive . In some embodiments , a liquid -resistant or liquid
fluid flow 760 flowing out of the servers 120 does not flow proof enclosure 892 for the movable hard -drive 890 in each
out of the outlet piping 750 . Instead , at least a portion of the of the servers 820 can be inserted over the hard drives prior
liquid coolant fluid flow 760 is through the heat exchanger 45 to the submersion of the server into the dielectric liquid
780 to the bottom of the tank 710 and then back through the coolant to protect moving components (e.g., a platen ) from
servers 120 . The heat rejected from the heat exchanger 780 being damaged by the viscous liquid coolant . The previously
is then thermally coupled to the second cooling fluid of the inserted liquid - proof enclosure traps air within the hard
second secondary cooling system for dissipation or recov -
drive . The entrapped air prevents the dielectric liquid cool
ery . 50 ant from entering the portion of the disk drive containing the
A combination of the system 100 and 200 using the movable disk .
alternative server rack apparatus of FIG . 9 and FIG . 10 that As shown in FIG . 13 , the apparatus 800 also may have
permit two different modes of operating the server rack cable trays 840 mounted along two sides of the tank 810
cooling system for cooling the dielectric liquid coolantmay paralleling the sides of the server racks 830 and 832 to
be useful in certain applications and climates, for example , 55 organize the signal and control network cabling 842 from the
in an arid climate having cool nights and very hot days . servers to the controller and other computers in the data
During the cool days , the combination system employing the center and beyond . The apparatus 800 may further have
embodiments of FIG . 9 or FIG . 10 may be used in a first power distribution units (“ PDUs” ) 844 mounted above the
mode similar to that of FIG . 1A wherein the dielectric fluid space between the server racks in order to distribute needed
is fluidly coupled to an external heat exchanger associated 60 electrical power through suitable power cables 846 to the
with a radiator -type secondary cooling system . During the multiple servers .
hot days , the combination system may be used in a second The server racks 830 and 832 may have a number of
mode similar to that of FIG . 1B wherein the dielectric liquid different implementations , some of which affect the flow
coolant is fluidly coupled through the internal heat characteristics of the liquid coolant. Preferably , the mount
exchanger, which is associated with a second secondary 65 ing members are configured to mountably receive the plu
cooling apparatus, such as a vapor - compression cycle refrig -
rality of servers in a vertical orientation , thereby minimizing
eration cooling system . the footprint of the servers relative to the ground , and with
US 10 , 123,463 B2
19 20
the “ front” 2 panel facing upward for easy installation and component temperatures below a maximum threshold . Fan
removal of a server without the need to remove or disturb speed can be modulated , but does not have to be .
any other server within the tank 810 . Additional fluid velocity augmentation devices, such as
- Upwards is defined as one of the two smallest sides of a rectangular server. multiple fans 880 may be mounted under each of the server
The “ back ” is generally referred to as the side with wires inserted , such as 5 racks 830 and 832 in the volume of liquid coolant between
power, communications, etc . the plurality of servers in each respective rack and the
As shown in FIGS. 12 and 14 , the mounting members bottom of the tank to increase the mixing of the dielectric
may be also configured to mount the servers such that the top liquid coolant within the tank , and improving the flow of the
level 824 of the liquid coolant 822 completely submerges the coolant through the plurality of servers . Other suitable fluid
top level 872 of the server rack 830 and 832 formed by the 10 augmentation devices include nozzles mounted on the end of
multiple servers 820. As a consequence , a volume of liquid a line from the cooling inlet piping which may be directed
coolant collects in a common manifold area above each of toward the desired entry point of the liquid coolant into the
the servers to improve the circulation of the liquid coolant servers to enhance the fluid velocity of the liquid coolant
through the plurality of servers , thereby enhancing the through the servers.
cooling of each respective server. The mounting members 15 FIG . 14 shows the fluid flow 860 of the liquid coolant 822
may also be configured to mount the servers in the server through the servers 820 in the apparatus 800 in more detail .
rack 830 and 832 above the bottom of the tank 810 to create For the server configuration shown , the fluid flow 860 of the
a volume of liquid coolant between each respective server piping liquid coolant entering the tank through the lower inlet
820 and the bottom of the tank such that the flow of the is initially directed through a volume of liquid
dielectric liquid coolant through the plurality of servers is 20 coolant 862 formed by the bottom of the side of the tank
improved . Preferably, the mounting members are configured rack containing the inlet piping and the bottom 872 of the server
to mount the servers closely adjacent to one another in the 830 and 832 of servers 820 and then through the bottom
server rack to restrict the flow of the dielectric liquid coolant 872 of the server rack through the servers 820 and out the top
between the plurality of vertically -oriented servers, such that 25 side 874 of the server rack into a volume of liquid coolant
876 formed by the top 874 of the server rack and the top
the flow of the dielectric liquid coolant through the plurality surface 822 of the liquid coolant and near the outlet piping
of servers is enhanced . located near the top of the tank .
The tank may also be sized and shaped to minimize the In summary , the immersion of servers into a liquid coolant
mixing of the cool and heated liquid coolant. Further the within the fixture apparatus various embodiments 400, 500 ,
apparatus 800 may include a removable top so that in the 30 600 , 700 , and 800 of the fixture apparatus shown in FIGS.
event of fire the top of the fixture apparatusmay be enclosed 3 - 14 reduces the temperature difference between server
to smother the fire . electronic components generating heat and the liquid coolant
A pump, such as the pump 330 of FIG . 2 , may pump medium used to cool them . Preferably, the median coolant
liquid coolant from the external heat exchanger through the temperature can be kept at as high a level as possible while
piping system into the tank 810 to maintain the coolant fluid 35 maintaining a component temperature during operation
flow within the tank. The liquid coolantmay flow through below its specified maximum allowable operating tempera
each installed server and exit through the outlet pipe from ture . Such a high -temperature cooling medium provides
the tank . Similar to FIGS. 3 thru 6 , the inlet piping may be sufficient cooling while reducing the power consumed to
located at one end of the rectangular tank 810 near the cool the electronic components, as compared to cooling the
bottom of the tank ; whereas the outlet piping may be located 40 component with a lower -temperature cooling medium such
nearer the top of the tank . This configuration permits the as refrigerated air .
liquid coolant heated by the heat generating components in Therefore the fixture apparatus for submerging the servers
the servers to naturally rise through the servers and exit in a dielectric liquid coolant provides for the following
through the front panel of the servers . Because the flow is advantages:
relatively low in comparison to the total volume of the 45 designed to maximize fluid temperature through flow
container , the fluid conducts to be relatively uniform tem control
perature . permits the use of standard commercially available rack
Alternatively , the location of the inlet and outlet piping mountable servers originally designed for air cooling
may be reversed such that the heated liquid coolantmay exit with minimal modification from commercially avail
from the installed servers through its “ rear” panel) into the 50 able configurations
outlet into the collection piping system . The collection transfers heat from all heat-generating components into
piping transports the heated liquid coolant to the heat the dielectric liquid coolant without the addition of cold
exchanger for rejecting at least some of the heat absorbed plates, piping or additional parts internal to the servers
from the installed servers . has an open top which enables the removal of any server
In commercially available servers , fans are often installed 55 without the removal of a different server (e .g ., servers
within the servers for distributing a cooling medium (e.g ., remain independently operable )
air ) among components and regions within the server. In only requires the tank enclosure to be sealed rather than
some embodiments , these fans can help distribute a liquid needing to hermetically seal each of the individual
coolant among the components and regions within the servers being mounted in the server racks
servers . Coolant flow rate and / or fan - speed can be adjusted 60 guides the fluid flow such that cool liquid coolant flows in
in response to a component temperature excursion above a and heated liquid coolant flows out of the servers
pre - determined threshold , or even a computational work may use fluid velocity augmentation , such as fan speed
load , to maintain component temperatures at or below a modulation , to enhance the flow of the liquid coolant
maximum specified (as by, for example , the component through each server
manufacturer ) temperature , while at the same time main - 65 improves the installed density of servers in a conventional
taining a coolant temperature at an elevated temperature , server room or data center by minimizing the footprint
such as at the highest coolant temperature that still maintains of the servers relative to the ground
US 10 , 123,463 B2
21 22
uses a controller (i) to monitor temperature and flow secondary cooling apparatus 352 , 354 , 356 , etc to optimize
conditions in the fixture apparatus and the power con - the secondary cooling apparatus to the environmental and
sumption of the servers and cooling system to minimize server rack conditions and control the amount ofheat being
the amount of power required to cool the servers and rejected by the selected secondary cooling apparatus by
(ii) to control the heat exchange method , thereby 5 adjusting the flow of the cooling fluid in the secondary
enabling the data center to recapture heat, if desirable , cooling apparatus.
or dissipate the heat in the most efficient manner when
heat recapture is not desirable . Methods of Operation
FIG . 15 depicts a schematic illustration of a system for
cooling multiple immersion - cooled server racks of the type 10 FIG . 16 illustrates an exemplary method of cooling one or
shown in , for example , FIG . 3 , located in a server room of more independently operable servers at least partially
a typical data center. The cooling system includes multiple immersed within a liquid coolant inside a tank with an open
server racks 310 fluidly coupled in parallel through respec interior volume. This method may be used to implement the
tive outlet piping 315 to collection piping system 902 . systems of FIG . 1A or 1B . The method includes a step 10 of
Collection piping 902 collects the heated liquid coolant 15 flowing a dielectric liquid coolant in a fluid circuit through
flowing out of the multiple server racks. The collection the plurality of servers immersed within the dielectric liquid
piping 902 , in turn , is fluidly coupled to a pump 904 which coolant for absorbing at least a portion of any heat being
pumps the collected heated liquid coolant through piping dissipated by the servers . In step 12 , the temperature of the
906 to a fluid line 908 in a heat exchanger 910 . The heated liquid coolant at at least one location is monitored by a
liquid coolant in fluid line 908 is thermally coupled to a 20 controller. In step 14 , the controller determines what tem
cooling fluid flowing in line 912 through heat exchanger perature would be the optimum elevated temperature of the
910 . The cooling fluid in line 912 , in turn , is coupled to a heated dielectric liquid coolant as it exits the plurality of
selected one of the heat rejection or cooling apparatus 352 servers such that the exiting liquid coolant sufficiently cools
354, 356 , etc as previously described for either dissipating or the plurality of servers while reducing the amountof energy
recovering the heat absorbed by the cooling fluid from the 25 consumed to sufficiently cool each respective server. As
heated liquid coolant. previously described , the determined optimum elevated tem
The cooled liquid coolant exiting from line 908 of the heat perature preferably is a temperature significantly higher than
exchanger 910 is then fluidly coupled through distribution the typical comfortable room temperature for humans and
piping system 914 to a plurality of parallel piping 916 fluidly lower than the maximum permissible temperature of the
connected to valves 918 . Valves 918 , in turn , are fluidly 30 most sensitive heat generating electronic component in the
connected in parallel to the inlet piping 370 to the multiple servers . In step 16 , the controller periodically determines the
server racks 310 . energy needed to reject the heat absorbed by the liquid
The controller 920 may receive monitoring signals of the coolant and maintain the liquid coolant exiting the servers at
temperature of the heated liquid coolant exiting the server the elevated temperature. In step 18 , the optimum secondary
racks through control lines 924 . The controller may also 35 cooling apparatus to minimize the amount of energy needed
receive monitoring signals of the flow rate of the liquid to be consumed to maintain the elevated temperature and
coolant at various locations in the piping 902 through cool the servers is selected . In step 20, the liquid coolant
control lines 925 and the flow rate through the pump 904 heated by the servers is thermally coupled to a heat
through control lines 926 . The controller 920 may also exchanger. In step 22 , a portion of the heat absorbed by the
receive monitoring signals relating to the type of secondary 40 liquid coolant from the servers is rejected through the heat
cooling apparatus selected and the flow rate of the cooling exchanger. In step 24 , in response to the energy consumption
fluid in the selected secondary cooling apparatus through periodically determined , the amount ofheat rejected through
control lines 928 . the heat exchanger is periodically adjusted such that the
As previously described , the controller 920 operates an liquid coolant exiting the plurality of servers at the elevated
application program that processes the information received 45 temperature sufficiently cools the plurality of servers while
from the variousmonitoring signals to selected an optimum reducing the amount of energy consumed to sufficiently cool
elevated temperature , the energy needed to be rejected by each respective server.
the system to cool the servers and maintain the elevated It should be noted that it may be desirable to also monitor
temperature , and then determine the various settings of the (i) the temperature of the liquid coolant atmultiple locations ,
system 900 components that will be needed to maintain the 50 ( ii ) the flow rate of the liquid coolant through the fluid
elevated temperature of the liquid coolant exiting the servers circuit; ( iii) the temperature of the electronic components of
in the multiple server racks 310 . The various components of the respective servers by connecting the temperature signals
the system 900 controlled by the controller 920 include any outputted by standard commercially available servers to the
fluid velocity augmentation devices positioned below the controller ; and the power consumption of the servers
server racks, the pump 904 , valves 918 , the valve 390 (FIG . 55 through signals outputted from the servers to the controller.
2 ) for switching the flow of the heated liquid coolant In response to the energy consumption periodically deter
between secondary cooling apparatus to be used , and the mined and the flow rate , the controller may periodically
selected secondary cooling apparatus . adjust the pumping rate of the liquid coolant through the
The controller may adjust the flow of the cooled liquid pump and the heat exchanger such that the liquid coolant
coolant through each of the valves 918 to adjust the volume 60 exiting the servers at the elevated temperature sufficiently
of the flow of the cooled liquid coolant among the different cools the plurality of servers while reducing the amount of
server racks 310 . energy consumed to sufficiently cool each respective server.
The controller 920 may control any fluid velocity aug . In connection with the operation of the cooling system
mentation devices in the server racks through control lines depicted in FIG . 1A and further depicted in FIG . 2 , the heat
and may also control the pumping rate of the pump 904 65 exchanger for directly rejecting heat from the liquid coolant
through control line 930 . In addition , the controller 920 is located externally to the fixture apparatus and the method
through control line 932 may select one of a plurality of employs a first type of thermodynamic cycle. In this embodi
US 10 , 123,463 B2
23 24
ment, the step of thermally coupling the liquid coolant to a flows into the tank with the servers . In step 26 , the dielectric
heat exchanger includes the step of fluidly coupling the liquid coolant flows in a fluid circuit through the plurality of
liquid coolant to a distally located heat exchanger and the servers immersed within the dielectric liquid coolant for
flow of the liquid coolant passes through outlet piping in the absorbing at least a portion of any heat being dissipated by
tank into a fluid circuit that is partially outside the tank . A 5 the servers . In step 28 , the fluid velocity of the liquid coolant
more detailed description of the steps occurring in this may be optionally enhanced by using fluid velocity aug
embodiment is set forth below in connection with the mentation devices, such as fans, in and outside of the
description of FIGS. 17A and 17B . servers . In step 30 , the temperature of the liquid coolant is
In connection with the operation of the cooling system monitored at least one location within the fluid circuit . In
200 in FIG . 1B , the coupler, such a heat exchanger, for 10 step 32 , a secondary cooling system is selected to minimize
directly rejecting heat from the heated liquid coolant flowing energy usage . In step 34 , the liquid coolant heated by the
through the servers 120 is located internally to the tank 210 . servers pumped to a heat exchanger distally located from the
The method of operation of this system 200 employs a tank . In step 36 , at least a portion of the heat absorbed by the
second type of thermodynamic cycle . In this alternative liquid coolant is rejected through the heat exchanger. In step
system embodiment, the method include the steps of flowing 15 38 , the cooled liquid coolant is fluidly coupled back to the
at least a portion of the cooler liquid coolant in a first fluid tank . In step 40 , the fluid flow in the secondary cooling
portion of a first liquid circuit through each of the plurality apparatus is adjusted to aid in maintaining the elevated
of servers wherein the liquid coolant exiting the plurality of temperature . In step 42, the rejected heat is dissipated
servers is heated to an elevated temperature ; thermally through the selected secondary cooling apparatus or in step
coupling the heated liquid coolant through a coupler to a 20 44, the rejected heat is recovered by the selected secondary
cooling fluid located in a first portion of a second fluid cooling apparatus .
circuit; fluidly coupling the heated cooling liquid in the first FIG . 17B illustrates the computer controller-based steps
portion of the second fluid circuit to an external distally in the method of cooling one or more independently oper
located heat exchanger for rejecting at least a portion of the able servers immersed in tank of liquid coolant employing
heat coupled through the second liquid circuit from the 25 the system of FIG . 1A or FIG . 3 . In step 52, the controller
heated dielectric liquid coolant; fluidly coupling the cooled receives signals relating to the system operation from vari
cooling fluid from the distally located heat exchanger o us sensors relating to temperature , fluid flow , and power
through a second portion of the second liquid circuit to the consumption . In step 54 , the controller determines the
coupler; thermally coupling the cooled cooling fluid through optimum elevated temperature for cooling the servers. In
the coupler to the first portion of the first liquid circuit . 30 step 56 , the controller periodically determines the energy
This method may also include the steps of monitoring the needed to cool the plurality of servers . In response to the
flow rate of the cooling fluid in the second fluid circuit ; and energy consumption periodically determined , the controller
monitoring the temperature of at least one of the heat - in step 58 periodically determining the optimal secondary
generating electronic components in each respective server; cooling method to minimize energy usage in order to adjust
periodically determining the energy needed to cool the 35 the amount of heat to be rejected through the heat exchanger
servers by the cooling of the heated cooling fluid to the such that the liquid coolant exiting the plurality of servers at
cooler temperature. This method may also include the step the elevated temperature sufficiently cools the plurality of
of enhancing the fluid velocity of the dielectric fluid through servers while reducing the amount of energy consumed to
the servers using fluid velocity augmentation devices, such sufficiently cool each respective server. In step 60 , the
fans or nozzles, as previously described herein . 40 controller determines the preferable settings for the dielec
In response to the controller periodically determining the tric liquid coolant pump , type of secondary cooling appa
energy needed to reject the absorbed heat and the flow rate ratus , and optionally the fluid velocity of the liquid coolant
of the cooling liquid , the method may also include the step in the tank . In step 62 , the controller executes the output
of periodically adjusting the flow rate of the cooling liquid control signals to the pumps, valves, and fluid velocity
through the second fluid circuit such that the liquid coolant 45 augmentation systems, i.e . fans or nozzles . In step 64 , the
exiting the servers at the elevated temperature sufficiently controller provides a failure notification in the event the
cools the servers while reducing the amount of energy system fails to operate as planned . For example the control
consumed to sufficiently cool each respective server . The ler provides a failure notification is there is a safety issue or
method may further include the steps of monitoring the the system is down for any reason .
temperature of the cooling fluid in the second fluid circuit. 50 In summary, the implementation of the methods disclosed
It should be noted that in the system employing the second in the exemplary alternate embodiments described herein for
type of thermodynamic cycle , the flow of the liquid coolant cooling server racks immersed in a dielectric liquid coolant
is contained inside the tank in which the servers are sub - by maintaining an elevated temperature can minimize the
merged . Preferably the fluid flow in this first fluid circuit is amount of power required to cool the servers . This accom
from the bottom of the server through the server to the top 55 plished by taking advantage of the number of irreversibili
thereof, where heated liquid coolant exists . Once the coolant ties or temperature differences present in a normal server
exits the top of the server , the coolant is cooled by passing cooling system that can be reduced to improve cooling
it through the heat exchanger in the liquid coolant. Once efficiency . The reduction of temperature differences between
cooled , the liquid coolant sinks to the bottom of the tank . the incoming cool liquid coolant and the heated outgoing
The flow of the coolant in the first fluid circuit can be 60 liquid coolant is made possible by:
supplemented by fans, internal or external to the servers . In controlling the amount of liquid coolant flow to each
the preferred embodiment, cooling takes place near the server by using speed modulated fluid velocity augmentation
exiting of the heated coolant from the servers. devices to ensure flow is sufficient to cool components with
FIG . 17A illustrates the physical steps in the method of changing demand; and
cooling one or more independently operable servers 65 a controller maintaining coolant temperature at the maxi
immersed in tank of liquid coolant employing the system of mum allowable temperature (e . g ., between 90 and 130
FIG . 1A or FIG . 3 . In step 24 of the method , liquid coolant degrees F .) by using the efficient heat removal methods
US 10 , 123 ,463 B2
25 26
described . The computer controller doesn 't necessarily have mountable server faces upward and the motherboard of
to separate from the servers that are being cooled . the rack -mountable server is vertically oriented ,
The reduction of irreversibilities in the thermodynamic wherein at least two of the rack -mountable servers are
cycle increases efficiency and therefore reduces overall independently operable from one another,wherein each
power consumed . With the described features, it should be 5 of at least two of the rack -mountable servers is con
possible to safely maintain fluid temperatures at approxi figured to connect to a computer network ;
mately 105 F, significantly higher than room temperature a volume of dielectric coolant at least partially held in the
and the maximum US average outdoor temperature by at least one tank ; and
month (75 degrees F . during summer ). At this temperature , a multi -circuit fluid cooling apparatus,
heat can be dissipated with minimum power or recaptured 10 wherein , when the vertically oriented rack -mountable
by heating other unrelated components such as building hot
water or ambient indoor air in cold climates . Further, this servers are mountably received in at least one row in
method should minimize or remove the need for energy the at least one tank , at least a portion of at least two of
intensive thermal processes associated with the current the vertically oriented rack -mountable servers being
methods of server/ computer cooling, which include refrig - 15 mountably received are held in the at least one tank
eration as the primary mode of heat dissipation . If heat such that the vertically oriented rack -mountable servers
dissipation (versus heat recapture ) is desired , an elevated are commonly at least partially submerged in the vol
coolant temperature allows methods requiring up to 1/8 or ume of the dielectric liquid coolant in the at least one
less power than conventional refrigeration methods. These tank for cooling the vertically oriented rack -mountable
low energy methods can include direct fluid to air heat 20 servers ,
exchangers , evaporative cooling, or other similar methods . wherein , when the vertically oriented rack -mountable
Refrigeration , however, can be used to supplement cooling servers are mountably received in the at least one tank
methods disclosed herein while consuming a minimum and commonly at least partially submerged in the
power. volume of dielectric liquid coolant in the at least one
Although the invention has been described with reference 25 tank , the vertically oriented rack -mountable servers are
to specific embodiments, these descriptions are not meant to thermally coupled to a remote heat exchanger via the
be construed in a limiting sense . Various modifications of the external fluid circuit, wherein the multi-circuit fluid
disclosed embodiments , as well as alternative embodiments cooling apparatus comprises :
of the invention will become apparent to persons skilled in a dielectric liquid cooling apparatus;
the art upon reference to the description of the invention . It 30 a secondary cooling apparatus comprising the remote
should be appreciated by those skilled in the art that the heat exchanger ; and
conception and the specific embodiment disclosed may be a liquid -to - liquid or liquid - to - refrigerant heat
readily utilized as a basis for modifying or designing other exchanger ;
structures for carrying out the same purposes of the present wherein the dielectric liquid cooling apparatus includes
invention . It should also be realized by those skilled in the 35 one or more pumps, and a plurality of fluid velocity
art that such equivalent constructions do not depart from the augmentation devices in the volume of dielectric liquid
spirit and scope of the invention as set forth in the appended coolant in the at least one tank , wherein at least one of
claims. the pumps, the coolant inlet , the coolant outlet , and the
It is therefore , contemplated that the claims will cover any mounting members are configured such that the at least
such modifications or embodiments that fall within the true 40 one pump moves at least a portion of the dielectric
scope of the invention . liquid coolant vertically across heat producing compo
What is claimed is : nents on the vertically oriented motherboard of one of
1 . An apparatus for cooling a plurality of rack -mountable the vertically oriented rack -mountable servers in par
servers containing heat generating electronic components , allel flow with dielectric liquid coolant moved verti
the cooling apparatus comprising: 45 cally across heat producing components on the verti
at least one tank defining an open interior volume and cally oriented motherboard of at least one other of the
having a coolant inlet for receiving a dielectric liquid vertically oriented rack -mountable servers,
coolant within the open interior volume and having a wherein each of at least two of the fluid velocity aug
coolant outlet for allowing the dielectric liquid coolant mentation devices is configured to increase upward
to flow from the open interior volume, the coolant inlet 50 fluid velocity of flow of dielectric liquid coolant
and the coolant outlet being fluidly coupled to each through the vertically oriented rack -mountable servers
other, wherein the at least one tank comprises an open or upwardly between adjacent ones of the vertically
or openable top sized to receive at least one row oriented rack -mountable servers ,
comprising a plurality of rack -mountable servers, each wherein the volume of dielectric liquid coolant comprises
of at least two of the plurality of rack -mountable 55 at least one passage in the tank that is outside of the
servers in the at least one row comprising a mother vertically oriented rack -mountable servers , wherein ,
board and heat producing components mounted to the when the at least one pump is operated to move the
motherboard ; dielectric liquid coolant vertically across the heat pro
one or more mounting members positioned within the ducing components on the vertically oriented mother
interior volume and configured to mountably receive 60 boards , a circuit is formed in which a first portion of
the plurality of rack -mountable servers within the inte dielectric liquid coolant is moved vertically upward
rior volume, wherein the mounting members are con across the heat producing components on the vertically
figured to hold each of at least two of the rack oriented motherboards and then downward outside of
mountable servers in the at least one row in a the rack mountable servers in the at least one passage ,
horizontally stacked relationship with one another, with 65 while a second portion of the dielectric liquid coolant
the rack -mountable servers in a vertical orientation flows out of the tank and through the liquid -to -liquid or
such that one of the two smallest sides of the rack liquid -to -refrigerant heat exchanger,
US 10 , 123,463 B2
28
wherein the liquid - to - liquid or liquid - to -refrigerant heat dently removable from the tank through the open top of
exchanger is configured to transfer heat from the sec the tank without the need to remove the other vertically
ond portion of dielectric liquid coolant in the circuit to oriented rack -mountable servers from the tank or dis
liquid or refrigerant moving in the secondary cooling turb the location or operational status of the other
apparatus , 5 vertically oriented rack -mountable servers remaining
wherein the secondary cooling apparatus is configured to within the tank .
reject, by way of the remote heat exchanger, at least 5 . The apparatus of claim 1 , wherein at least one of the
part of the heat transferred from the second portion of vertically oriented rack -mountable servers is vertically
the dielectric liquid coolant, removable from the at least one tank without the need to
wherein the vertically oriented rack -mountable servers 10 disturb the operational status of the other vertically oriented
are mountable in the at least one tank such that, when rack -mountable servers remaining within the at least one
the vertically oriented rack -mountable servers are com - tank .
monly at least partially submerged in the volume of 6 . The apparatus of claim 1 , wherein at least one of the
dielectric liquid coolant in the at least one tank and the vertically oriented rack -mountable servers is vertically
dielectric liquid coolant is moved within the at least one 15 removable from the at least one tank without the need for
tank , at least one of the vertically oriented rack -mount- horizontal detachment of the at least one rack -mountable
able servers is independently vertically removable from server.
the volume of dielectric liquid coolant in which the at 7 . The apparatus of claim 1, wherein at least one of the
least two vertically oriented rack -mountable servers are vertically oriented rack -mountable servers comprises an
commonly at least partially submerged and from the at 20 enclosure , wherein flow of the dielectric liquid coolant
least one tank through the top of the tank while at least around the vertically oriented rack -mountable servers is
two of the other vertically oriented rack -mountable restricted such that flow through the enclosure of the at least
servers in the at least one row remain commonly at least one vertically oriented rack -mountable server is enhanced .
partially submerged in the volume of dielectric liquid 8 . The apparatus of claim 1 , wherein one or more of the
coolant in the at least one tank and the dielectric liquid 25 vertically oriented rack -mountable servers are not attached
coolant in the at least one tank continues to flow across to a lid for the apparatus .
heat producing electronic components of the at least 9 . The apparatus of claim 1, wherein the one or more
two other vertically oriented rack -mountable servers mounting members comprise :
remaining at least partially commonly submerged in the a first set of one or more mounting members coupled in
at least one row . 30 a fixed relationship to the tank ; and
2 . The apparatus of claim 1, wherein : a second set of one or more mounting members coupled
the one or more mounting members are configured to in a fixed relationship to the tank ,
mountably receive the vertically oriented rack -mount wherein the first set of one or more mounting members
able servers above the bottom of the at least one tank and the second set of mounting members are spaced
to form a volume between each respective vertically 35 apart from one another to form at least one opening for
oriented rack -mountable server and the at least one tank receiving a row of two or more of the vertically
to permit the flow of dielectric liquid coolant through oriented rack -mountable servers in a vertical orienta
the plurality of vertically oriented rack -mountable serv tion within the interior volume, wherein the at least one
ers . mounting member in the first set of mounting members
3 . The apparatus of claim 1 , wherein : 40 supports at least one vertically oriented rack -mountable
the vertically oriented rack -mountable servers have a server of the vertically oriented rack -mountable servers
plurality of rack ears ; on one side of the row and wherein at least one
the at least one tank comprises a tank which is sized to mounting member in the second set ofmounting mem
receive at least one row of the vertically oriented bers supports the at least one vertically oriented rack
rack -mountable servers ; and 45 mountable server on the other side of the row .
the one or more mounting members comprises a plurality 10 . The apparatus of claim 1 , wherein the plurality of fluid
of mounting members fixedly attached to the at least velocity augmentation devices comprise a plurality of
one tank and configured to mountably receive the rack nozzles configured to direct dielectric liquid coolant from
ears of the plurality of vertically oriented rack -mount- the coolant inlet piping toward a desired entry point into the
able servers in the at least one row of vertically oriented 50 vertically oriented rack -mounted servers to augment the
rack -mountable servers to permit installation and fluid velocity of the liquid coolant through the vertically
removal of the at least one vertically oriented rack oriented rack -mounted servers .
mountable server without affecting the operational sta 11 . The apparatus of claim 10 , further comprising a line
tus of the other vertically oriented rack -mountable from the coolant inlet, wherein the plurality of nozzles are
servers remaining within the at least one tank . 55 mounted on the line, wherein each of at least two of the
4 . The apparatus of claim 1 , further comprising: plurality ofnozzles are configured to direct dielectric liquid
at least one cable tray to receive and support signal and coolant toward a desired entry point into the vertically
control network cabling to and from the plurality of oriented rack -mounted servers to augment the fluid velocity
vertically oriented rack -mountable servers ; and of the liquid coolant through the vertically oriented rack
at least one power distribution unit to receive and support 60 mounted servers .
power distribution cables for distributing electrical 12 . The apparatus of claim 1, wherein at least two of the
power to the plurality of vertically oriented rack - fluid velocity augmentation devices are mounted in the
mountable servers ; volume of dielectric liquid coolant between the at least one
wherein the at least one cable tray and the at least one row of vertically oriented rack -mountable servers and bot
power distribution unit are mounted in proximity to the 65 tom of the at least one tank such that the fluid velocity
open top of the at least one tank so that the at least one augmentation devices increase upward flow of the dielectric
vertically oriented rack -mountable server is indepen - liquid coolant through or between the plurality of vertically
US 10 , 123 , 463 B2
29 30
oriented rack -mountable servers , wherein each of at least mountable servers in the at least one row in a
two of the fluid velocity augmentation devices is configured horizontally stacked relationship with one another, with
to increase upward fluid velocity of flow of dielectric liquid the rack -mountable servers in a vertical orientation
coolant in a different part of the volume of dielectric liquid such that one of the two smallest sides of the rack
coolant in which the at least two vertically oriented rack 5 mountable server faces upward and the motherboard of
mountable servers are commonly at least partially sub the rack -mountable server is vertically oriented ,
merged and upwardly through different ones of the vertically wherein at least two of the rack -mountable servers are
oriented rack -mountable servers or upwardly through a gap independently operable from one another , wherein each
between different adjacent pairs of the fluid velocity aug of at least two of the rack -mountable servers is con
mentation devices rack -mountable servers . 10 figured to connect to a computer network ,
13. The apparatus of claim 1, wherein at least one of the a volume of dielectric coolant at least partially held in the
one or more pumps is configured to move a portion of the at least one tank ; and a multi- circuit fluid cooling
dielectric liquid coolant across heat producing electronic apparatus ,
components of the other vertically oriented rack -mountable wherein , when the vertically oriented rack -mountable
servers while the at least one of the plurality of vertically 15 servers are mountably received in at least one row in
oriented rack -mountable servers is removed from the vol the at least one tank , at least a portion of at least two of
ume of dielectric liquid coolant. the vertically oriented rack -mountable servers being
14 . The apparatus of claim 1, wherein , when the vertically mountably received are held in the at least one tank
oriented rack -mountable servers are mountably received in such that the vertically oriented rack -mountable servers
the at least one tank and the dielectric liquid coolant is 20 are commonly submerged in the volume of the dielec
moved within the at least one tank , heated dielectric liquid tric liquid coolant in the at least one tank for cooling the
coolant at the surface of the volume of dielectric liquid vertically oriented rack -mountable servers,
coolant is moved crossways relative to the direction of the wherein , when vertically oriented rack -mountable servers
row of vertically oriented rack -mountable servers . are mountably received in the at least one tank and
15 . The apparatus of claim 1, further comprising a con - 25 commonly at least partially submerged in the volumeof
troller configured to : dielectric liquid coolant in the at least one tank, the
monitor one or more operational parameters of cooling of vertically oriented rack -mountable servers are ther
the vertically oriented rack -mountable servers ; mally coupled to a remote heat exchanger outside of a
determine, based at least in part on at least one of the server room in which the at least one tank is located via
monitored operational parameters , an elevated tem - 30 the multi- circuit fluid cooling apparatus, wherein the
perature of dielectric liquid coolant exiting the verti multi-circuit fluid cooling apparatus comprises:
cally oriented rack -mountable servers , wherein the a dielectric liquid cooling apparatus;
elevated temperature is selected to sufficiently cool a secondary cooling apparatus comprising a remote
each of the vertically oriented rack -mountable servers heat exchanger ; and
while reducing the total amount of energy need to cool 35 a liquid - to -liquid or liquid -to -refrigerant heat
the vertically oriented rack -mountable servers ; and exchanger ;
maintain dielectric liquid coolant exiting the vertically wherein the dielectric liquid cooling apparatus includes
oriented rack -mountable servers at the selected one or more pumps , and a plurality of fluid velocity
elevated temperature . augmentation devices in the volume of dielectric liquid
16 . The apparatus of claim 15 , wherein the controller is 40 coolant in the at least one tank , wherein at least one of
configured to operate at least one of the one or more pumps the pumps, the coolant inlet, the coolant outlet, and the
to maintain dielectric liquid coolant exiting the vertically mounting members are configured such that the at least
oriented rack -mountable servers at a temperature that is one pump moves at least a portion of the dielectric
significantly higher than comfortable room temperature and liquid coolant vertically across heat producing compo
lower than the maximum permissible temperature of the 45 nents on the vertically oriented motherboard of one of
most sensitive heat generating electronic component of the the vertically oriented rack -mountable servers in par
vertically oriented rack -mountable servers . allel flow with dielectric liquid coolant moved verti
17 . The apparatus of claim 1 , wherein the liquid -to -liquid cally across heat producing components on the verti
heat exchanger or refrigerant-to - liquid heat exchanger is cally oriented motherboard of at least one other of the
located on the at least one tank . 50 vertically oriented rack -mountable servers ; and
18 . An apparatus for cooling a plurality of rack -mountable wherein each of at least two of the fluid velocity aug
servers containing heat generating electronic components in mentation devices is configured to increase upward
a server room , the cooling apparatus comprising : fluid velocity of flow of dielectric liquid coolant
at least one tank defining an open interior volume and through the vertically oriented rack -mountable servers
having a coolant inlet for receiving a dielectric liquid 55 or upwardly between adjacent ones of the vertically
coolant within the open interior volume and having a oriented rack -mountable servers,
coolant outlet for allowing the coolant to flow from the wherein the volume of dielectric liquid coolant comprises
open interior volume, the coolant inlet and the coolant at least one passage in the tank that is outside of the
outlet being fluidly coupled to each other, each of at vertically oriented rack -mountable servers, wherein ,
least two of the plurality of rack -mountable servers in 60 when the at least one pump is operated to move the
the at least one row comprising a motherboard and heat dielectric liquid coolant vertically across the heat pro
producing components mounted to the motherboard ; ducing components on the vertically oriented mother
one or more mounting members positioned within the boards, a circuit is formed in which a first portion of
interior volume and configured to mountably receive dielectric liquid coolant is moved vertically upward
the plurality of rack -mountable servers within the inte - 65 across the heat producing components on the vertically
rior volume, wherein the mounting members are con oriented motherboards and then downward outside of
figured to hold each of at least two of the rack the rack mountable servers in the at least one passage ,
US 10 , 123,463 B2
32
while a second portion of the dielectric liquid coolant wherein at least two of the rack -mountable servers are
flows outof the tank and through the liquid -to -liquid or independently operable from one another, wherein each
liquid - to - refrigerant heat exchanger , of at least two of the rack -mountable servers is con
wherein the liquid - to - liquid or liquid - to - refrigerant heat figured to connect to a computer network ,
exchanger is configured to transfer heat from the sec - 5 a volume of dielectric coolant at least partially held in the
ond portion of dielectric liquid coolant in the circuit to at least one tank ; and
liquid or refrigerant moving in the secondary cooling a multi -circuit fluid cooling apparatus,
apparatus, wherein , when the vertically oriented rack -mountable
wherein the secondary cooling apparatus is configured to servers are mountably received in at least one row in
reject, by way of the remote heat exchanger, at least 10 the at least one tank , at least a portion of at least two of
part of the heat transferred from the second portion of the vertically oriented rack -mountable servers being
the dielectric liquid coolant. mountably received is held in the at least one tank such
19 . The apparatus of claim 18 , wherein , when the plurality that at least two of the vertically oriented rack -mount
of vertically oriented rack -mountable servers are mountably able servers can be commonly at least partially sub
received , each respective rack -mountable server is com - 15 merged in the volume of the dielectric liquid coolant in
pletely submerged within the dielectric liquid coolant such the at least one tank for cooling the vertically oriented
that a volume of dielectric liquid coolant collects in a rack -mountable servers,
common manifold area above the plurality of rack -mount wherein , when the vertically oriented rack -mountable
able servers to improve the circulation of the liquid coolant servers are mountably received in the at least one tank
through or between the plurality of vertically oriented rack - 20 and commonly at least partially submerged in the
mountable servers , thereby enhancing the cooling of the volume of dielectric liquid coolant in the at least one
vertically oriented rack mountable servers . tank , the vertically oriented rack -mountable servers are
20 . The apparatus of claim 18 , wherein the at least one thermally coupled to a remote heat exchanger outside
tank comprises an open or openable top , wherein the verti of the server room via the multi -circuit fluid cooling
cally oriented rack -mountable servers are mountable such 25 apparatus, wherein themulti - circuit fluid cooling appa
that at least one of the vertically oriented rack -mountable ratus comprises :
servers can be removed from the at least one tank through a dielectric liquid cooling apparatus; and
the open or openable top while at least two of the other a secondary cooling apparatus comprising a remote
vertically oriented rack -mountable servers remain com heat exchanger,
monly at least partially submerged in the dielectric liquid 30 wherein the dielectric liquid cooling apparatus includes
coolant and dielectric liquid coolant continues to flow across one or more pumps, and a plurality of fluid velocity
the at least two other vertically oriented rack -mountable augmentation devices in the volume of dielectric liquid
servers . coolant in the at least one tank , wherein at least one of
21 . The apparatus of claim 18 , further comprising a the pumps, the coolant inlet, the coolant outlet, and the
controller configured to maintain dielectric liquid coolant 35 mounting members are configured such that the at least
exiting the vertically oriented rack -mountable servers at a one pump moves at least a portion of the dielectric
temperature that is significantly higher than comfortable liquid coolant in the volume of dielectric liquid coolant
room temperature and lower than the maximum permissible vertically across heat producing components on the
temperature of the most sensitive heat generating electronic vertically oriented motherboard of one of the vertically
component of the vertically oriented rack -mountable serv - 40 oriented rack -mountable servers in parallel flow with
ers . dielectric liquid coolant moved vertically across heat
22 . An apparatus for cooling a plurality of rack -mountable producing components on the vertically oriented moth
servers containing heat generating electronic components in erboard of at least one other of the vertically oriented
a server room , the cooling apparatus comprising : rack -mountable servers ,
at least one tank defining an open interior volume and 45 wherein the vertically oriented rack -mountable servers
having a coolant inlet for receiving a dielectric liquid are mountable in the at least one tank such that, when
coolant within the open interior volume and having a the vertically oriented rack -mountable servers are com
coolant outlet for allowing the coolant to flow from the monly at least partially submerged in the volume of
open interior volume, the coolant inlet and the coolant dielectric liquid coolant in the at least one tank and the
outlet being fluidly coupled to each other, wherein the 50 dielectric liquid coolant ismoved within the at least one
at least one tank comprises an open or openable top tank , at least one of the vertically oriented rack -mount
sized to receive at least one row comprising a plurality able servers is independently vertically removable from
of rack -mountable servers, each of at least two of the the volume of dielectric liquid coolant in which the at
plurality of rack -mountable servers in the at least one least two vertically oriented rack -mountable servers are
row comprising a motherboard and heat producing 55 commonly at least partially submerged and from the at
components mounted to the motherboard ; least one tank through the top of the tank while at least
one or more mounting members positioned within the two of the other vertically oriented rack -mountable
interior volume and configured to mountably receive servers in the at least one row remain commonly at least
the plurality of rack -mountable servers within the inte partially submerged in the volume of dielectric liquid
rior volume, wherein the mounting members are con - 60 coolant in the at least one tank and the dielectric liquid
figured to hold each of at least two of the rack coolant in the at least one tank continues to flow across
mountable servers in the at least one row in a heat producing electronic components of the at least
horizontally stacked relationship with one another, with two other rack -mountable servers remaining at least
the rack -mountable servers in a vertical orientation partially commonly submerged in the at least one row ,
such that one of the two smallest sides of the rack - 65 wherein each of at least two of the fluid velocity aug
mountable server faces upward and the motherboard of mentation devices is configured to increase upward
the rack -mountable server is vertically oriented , fluid velocity of flow of dielectric liquid coolant
US 10 , 123 ,463 B2
33 34
through the vertically oriented rack -mountable servers wherein at least two of the vertically oriented rack
or upwardly between adjacent ones of the vertically mountable servers are independently operable from one
oriented rack -mountable servers , another, wherein each of at least two of the vertically
wherein the volume of dielectric liquid coolant comprises oriented rack -mountable servers is configured to con
at least one passage in the tank that is outside of the 5 nect to a computer network ,
vertically oriented rack -mountable servers , wherein , a volume of dielectric coolant at least partially held in the
when