US10405457B2 — Appliance immersion cooling system
Document text
Research, not advice. Part of the Bitcoin research archive (October 2026). Claims labelled unverified, contested or fringe are reported, not endorsed; statuses of bills and rules are as of the date checked. Government, court and patent records are public domain; the research notes are CC BY 4.0.
US010405457B2
(12) United
Boyd et al.
States Patent ( 10) Patent No.: US 10 ,405 ,457 B2
(45) Date of Patent: Sep . 3 , 2019
(54 ) APPLIANCE IMMERSION COOLING 1) Int. CI.
SYSTEM HOIL 23/44 (2006 .01)
H05K 7/ 20 ( 2006 .01)
(71 ) Applicants :Christopher L . Boyd , Austin , TX (US ); 2 ) U . S . CI.
James P . Koen , Round Rock , TX (US ); CPC ......... H05K 7/ 20236 ( 2013. 01 ); H01L 23/ 44
David Christopher Laguna , Austin , (2013. 01 ); H05K 7/ 20272 (2013 .01)
TX (US ); Thomas R . Turner, (58 ) Field of Classification Search
Georgetown, TX (US); Kenneth D . CPC ...................... HO5K 7 / 20236 ; HO5K 7 /20272;
Swinden , Hutto , TX (US ); Mario HO1L 23 /42 ; HO1L 23/ 44
Conti Garcia , Austin , TX (US) ; John (Continued )
Charles Tribou , Austin , TX (US )
(72 ) Inventors: Christopher L . Boyd , Austin , TX (US); (56) References Cited
James P . Koen , Round Rock , TX (US ); U .S . PATENT DOCUMENTS
David Christopher Laguna, Austin ,
TX (US) ; Thomas R . Turner , 4 ,590,538 A * 5/1986 Cray, Ir. .......... HO5K 7361/20236
Georgetown, TX (US ); Kenneth D . 5, 167,511 A * 12/1992 Krajewski ............ HO1R
/700
4 /01
Swinden , Hutto , TX (US ); Mario 361/ 785
Conti Garcia , Austin , TX (US ); John
Charles Tribou , Austin , TX (US) (Continued )
( 73 ) Assignee : Midas Green Technologies, LLC , FOREIGN PATENT DOCUMENTS
Austin , TX (US) ... G06F 1/20
JP 5956100 B1 * 7 /2016
( * ) Notice: Subject to any disclaimer, the term of this RU 2042294 Ci 8 / 1995
patent is extended or adjusted under 35 SU 1764094 AL 9 /1992
U .S .C . 154 (b ) by 680 days . Primary Examiner — Devon Russell
( 21) Appl. No.: 14 /355 ,533 (74 ) Attorney, Agent, or Firm — Jeffrey Van Myers
(22 ) PCT Filed : Dec . 13 , 2013 (57) ABSTRACT
(86 ) PCT No.: PCT/US2013/075126 A appliance immersion tank system comprising : a generally
rectangular tank adapted to immerse in a dielectric fluid a
$ 371 (c )( 1), plurality of appliances, each in a respective appliance slot
( 2 ) Date : Apr. 30 , 2014 distributed vertically along, and extending transverse to , the
long axis of the tank; a primary circulation facility adapted
(87 ) PCT Pub. No.: WO2014 /109869 to circulate the dielectric fluid through the tank ; a secondary
PCT Pub. Date: Jul. 17, 2014 fluid circulation facility adapted to extract heat from the
dielectric fluid circulating in the primary circulation facility ,
(65) Prior Publication Data and to dissipate to the environment the heat so extracted ; and
US 2015 /0181762 A1 Jun . 25 , 2015 a control facility adapted to coordinate the operation of the
primary and secondary fluid circulation facilities as a func
Related U .S. Application Data tion of the temperature of the dielectric fluid in the tank . A
(60 ) Provisional application No.61/737 ,200 , filed on Dec . plenum , positioned adjacent the bottom of the tank , is
14 , 2012, provisional application No. 61/ 832 ,211 , adapted to dispense the dielectric fluid substantially uni
filed on Jun . 7 , 2013 . (Continued )
34b
706 Wh
34a y
ww w w
- 72
36 Il 70a
US 10 ,405 , 457 B2
Page 2
formly upwardly through each appliance slot. A weir, inte
grated horizontally into a long wall of the tank , is adapted to
facilitate substantially uniform recovery of the dielectric
fluid flowing through each appliance slot. All active and
most passive components ofboth the primary and secondary
fluid circulation facilities, and the control facility are fully
redundant, and are adapted automatically to operate in a
fail - soft mode .
16 Claims, 7 Drawing Sheets
(58 ) Field of Classification Search
USPC ............. 361/699
See application file for complete search history.
(56 ) References Cited
U .S . PATENT DOCUMENTS
5 ,297 ,621 A * 3/ 1994 Taraci ............... GO1R 31/ 2891
165/ 104 .13
8 ,009 ,419 B2 8/ 2011 Attlesey et al.
2005/0259402 A1 * 11/ 2005 Yasui HO2M 7/003
361/716
2006 /0126292 A1 * 6 / 2006 Pfahnl ............ H05K 7 /20563
361/695
2006 /0274501 A1* 12 /2006 Miller .. ............. GOIR 31/ 2863
361/690
2011/0075353 A1 * 3 / 2011 Attlesey .............. G06F 1 / 20
361/679 .47
2011/0132579 A1 * 6 /2011 Best . . ... .. .. .. .. .. ... . HO5K 7 /20763
165 / 104.31
2011/0240281 A1 * 10 / 2011 Avery ............. GO5D 23 / 1917
165 /287
* cited by examiner
U . S . Patent Sep . 3 , 2019 Sheet 1 of 7 US 10 ,405,457 B2
34b
W W W W W W W W W WW W W ?
.
ir
'
Yawa w wm m m Inwong
.
34a
34a INC
20 .WWW
t
l.
bit :1 .1
WAittlig
. 51:161.1
in .1. . 11
:
Event
1
A
.
ti 1
orenwon ww w w w w ww
YwanerUYUYLYWYriririririririririri
:21
9 :u
10 .
55 t1.1 11 16:) ** nu
L
LUB.1
.
Hurumu
.t.run 12
.H . .,1
11
11
. .1..Thef
L1. a t.
.11.
*114.164 ..1tit
0.4.1.r
20 1
.
11. The .1 .1:1
9. 11 14
10: 11 .
14 .1
WW .1 .117
11 .14: 11.
215.: Writi 13..14.11
DOH
10
72
W
tinin Vanuit.195. .011 70a
4116.10 1-11Tinit 1
1. . .1.14.MAU
we r
m
1 .
.
15 : 10yu : 15
! .
MAX
.
4
10 eur
14
Fig. 1
U . S . Patent Sep . 3 , 2019 Sheet 2 of 7 US 10 , 405 ,457 B2
34b
46
MAWKNUALIWD.AHOULUCNULUKUTA
38 Unimin
n
a
w w w ww w
COISpAKoCSrecucokurckOoCKcSMIGOoS
rweuwsrtwamluremamnetneatn
RINNEN
!with
APUNAR UHA $
w hite
34a
AP Santa 320
GAMAKANKcAKBOUhLULAiUGUnLAKAiNG 486286
33
32a
Fig.2 38
46 temu
44a
.7
48a > 28a
4
:2
.W
. 21
fratering
'
n
93
Ex
**
* VV2 ww w w 4
Wow.w
4
.
27
inhet
ht hetlicht
w
Fig.4 CALE ): 48a 42 setmanesen Het Fig . 3
U . S . Patent Sep . 3 , 2019 Sheet 3 of 7 US 10 ,405 ,457 B2
.
S
*
2
.
*
*
**
*
*
*
w * ** * the
SA mm 22
SOULAN
M w
* * ** * ** ** **
Fig . 5
WWW www
NNN WWW .
14
Section C -C
Fig . 6
U . S . Patent Sep . 3 , 2019 Sheet 4 of 7 US 10, 405,457 B2
.
BUWIE
.
n 1
TUP
VN .HR
ya
TwWEMA
1
WiiwtiN.
Wit u . I' I. :4.. VMI
WWW. NO OM ie
4,
.
W h
1
.. 11
WWW2:.bebu . 2 Wit .. : tu
H
huNotturn
W13. .... 1.
1.
1.
AURI
0
21
Hup
WWW
IV . . 1.
.
Pattur
n.
m . tw
-H WIN 1
2.LLA
ANOPY
-1:..
ritud
1. 1 . *..
.1
N.1,PRW
* .1
.
VW 4 O
8
!
... W hil
...
thp
h2
. hr,RN1
# 0 .94
1
.
Wt NYIM.Mhht .
.
N. P. W 1
* * **
#AMFOAI
NINNU. .WK16
U W
WRt
2. ru 4
Niti LUKA *
**
Sport
Fig. 7
HEIM
WWW.
W O
:
.
1 41NAVA
I
MMK3
W*
0
. 11
..
.4:1
*UWh
. N
A
T *
*
*
#
A
#
36a
# .
tt *
.
4
Y
#
ut 3
OU
ts
Fig. 8
36b
Fig . 9
U . S . Patent Sep . 3 , 2019 Sheet 5 of 7 US 10 ,405 ,457 B2
??
Fig. 10
16 c 22 14a 68 14b
+ ". . . . .. . .. . . . -. I
2
21 : 1 :
O BEX
.
* 2
M M M
XXX X AW WW
WMWWWW
* * * * * * * * ** * * * * * * * **
X X X * * * ** * *WWW *
KWAKUWA
NWWLWWLISISSA SISSE
2
36 Fig . 11 66
U . S . Patent Sep . 3 , 2019 Sheet 6 of 7 US 10 ,405,457 B2
14
C
36
*
*
.WIONOXXU IKKE
- WWWXXXXXXXX
44b
48a 48b 38
28a 28b
IIIIIIIIIII
30 32a 32b
52b
50b
30a 54a 1 30b 546 11
Fig . 12
U . S . Patent Sep . 3 , 2019 Sheet 7 of 7 US 10 ,405,457 B2
22
14 *
86
*
- -- -
-- ---- -- ---
Primary
Controller
58a
48a
28a 4 4
.. RRRRRRRARA D IDINIEU Master /
TCP
IP
.
30a
Controller
32a
NOON . JOJO ,
M
Secondary
Controller
60a
Tower
50a i
Fig . 13
US 10 ,405 ,457 B2
APPLIANCE IMMERSION COOLING is an essential element in the system architecture , such as the
SYSTEM central processing unit (“ CPU ” ) . One possible solution to
this problem is to immerse circuit assemblies vertically into
CROSS -REFERENCE TO RELATED a tank containing the cooling fluid such that each of the
APPLICATIONS 5 various assemblies can be withdrawn independently from
the tank for servicing , replacement, upgrade , etc. One inter
This application is related to the following Provisional esting example of such a system is disclosed in a web
Applications : presentation entitled “ Puget Custom Computer 's mineral
1 . Ser. No. 61/737 ,200, filed 14 Dec . 2012 (“ First Parent oil-cooled PC ” , by Nilay Patel (“ Puget” ) (posted 12 May
Provisional” ); and 2007 at 11 :57 AM ; a copy ofwhich is submitted herewith ).
2 . Ser . No . 61/832, 211, filed 7 Jun . 2013 (“ Second Parent As noted by the author, the lack of supplemental apparatus
Provisional" ); in the Puget system to extract waste heat from the oil
and hereby claimsbenefit of the filing dates thereof pursuant
to 37 CFR § 1.78 (a )(4 ). (Collectively , “ Parent Provision inherently limited its operating capabilities .
als” ). The subject matter of the Parent Provisionals , each in 15 particular is problem
Another
the
with the Cray Research systems in
nature and cost of the chosen cooling fluid :
its entirety , is expressly incorporated herein by reference .
fluorocarbon liquids. As is known, other dielectric fluids ,
BACKGROUND OF THE INVENTION such as mineral oil, have better heat transfer characteristics;
of course , being an oil , the use thereof does represent a
1. Field of the Invention 20 greater residue problem on modules that may be repairable .
The present invention relates generally to electrical appli - Notwithstanding , the Puget system implemented precisely
ance cooling systems, and , in particular, to an improved this design choice .
appliance immersion cooling system and method of opera US Patent Application Publication 2011/0132579, “ Liq
tion . uid Submerged , Horizontal Computer Appliance Rack and
2 . Description of the Related Art 25 Systems and Method of Cooling such a Appliance Rack ” ,
In general, in the descriptions that follow , we will italicize Best, et al. (“ Best” ), discloses a appliance immersion tank
the first occurrence of each special term of art which should system , include support apparatus for extracting waste heat
be familiar to those skilled in the art of immersion cooling from the tank cooling fluid and dissipating to the environ
systems. In addition , when we first introduce a term that we ment the heat so extracted . Although an improvement in
believe to be new or that we will use in a context that we 30 several respects over the prior art discussed above , this
believe to be new , we will bold the term and provide the system exhibits , inter alia , the following problems: generally
definition that we intend to apply to that term . non -uniform flow patterns through the several appliance
U . S . Pat. No. 4 , 590 ,538 , “ Immersion Cooled High Den slots within the tank , potentially resulting in uneven cooling
sity Electronic Assembly ” , Cray ( filed 18 Nov . 1981 and across all slots ; constricted dielectric fluid supply and return
issued 20 May 1986 ) (“ Cray ” ), is an early example of an 35 ports resulting in unnecessarily high fluid flow velocities at
immersion system for cooling electronic components during the respective points of connection to the tank ; poor scal
normal operation . On information and belief, the machine ability ; and inadequate attention to fail- soft operation .
disclosed therein was the Cray - 2 super -computer (“ Cray - 2” ) The subject matter of all of the prior art references
manufactured by Cray Research , Inc . (“ Cray Research " ), of discussed above, each in its entirety, is expressly incorpo
Chippewa Falls, Wis. Of particular interest to the present 40 rated herein by reference .
application is the description of the significant advantages We submit that what is needed is an improved appliance
resulting from using an electrically non -conductive or tank immersion system and method of operation . In particu
dielectric fluid to extract heat from electronic circuit assem lar, we submit that such a system should provide perfor
blies during normal operation ( see, e.g ., col. 1, line 66 -col. mance generally comparable to the best prior art techniques
2 , line 29 ). 45 but more efficiently and effectively than known implemen
On information and belief, Cray Research released , in tations of such prior art techniques .
1985 , a marketing brochure entitled “ The CRAY -2 Com
puter System ” ( a copy of which is submitted herewith ) BRIEF SUMMARY OF THE INVENTION
describing the Cray - 2 . Of particular interest in this brochure
is the description therein of the significant advantages result- 50 In accordance with a preferred embodiment of our inven
ing from using a dielectric fluid to extract heat from elec - tion , . . .
tronic circuit assemblies during normal operation (see, pages
10 and 13 ). BRIEF DESCRIPTION OF THE SEVERAL
U .S . Pat. No. 5 , 167,511 , “High Density Interconnect VIEWS OF THE DRAWINGS
Apparatus” , Krajewski, et al. ( issued 27 Nov. 1992) (“ Kra - 55
jewski” ), discloses another example of an immersion system Our invention may be more fully understood by a descrip
for cooling electronic components during normal operation tion of certain preferred embodiments in conjunction with
(see , e .g., col. 2 , lines 43 -51 ). On information and belief, a the attached drawings in which :
machine implementing the Krajewski system was also mar - FIG . 1 illustrates , in partial cut- away form , a front per
keted by Cray Research as a follow -on super- computer to 60 spective of a tank module of an appliance immersion cooling
the Cray -2 . system constructed in accordance with our invention ;
One particular problem in the vertical- stack -type systems FIG . 2 illustrates a rear perspective of the tank module
disclosed in the above references is the necessity of draining shown in FIG . 1 ;
the cooling fluid whenever physical access to the electronic FIG . 3 illustrates a close -up perspective of a detail A of
modules was required . In general, such an operation , besides 65 FIG . 2 ;
being time consuming , requires the entire system to be FIG . 4 illustrates a close - up perspective of a detail B of
switched off, especially if the component requiring attention FIG . 2 ;
US 10 ,405 ,457 B2
FIG . 5 illustrates, in perspective view , several details of shown in FIG . 2 ); and control equipment cabinets 34a and
the tank shown in FIG . 1 , with special emphasis on the 34b , each adapted to accommodate the module status and
dielectric fluid recovery weir integrated into the long rear control equipment associated with a respective one of the
wall of the tank ; primary circulation facilities 28a and 28b (see , FIG . 13 ).
FIG . 6 illustrates, in cross -section view , the section C C- 5 As can be best seen in FIG . 2 , the primary circulation
in FIG . 5 ; facility 28 (comprising redundant sub - facilities 28a and 28b )
FIG . 7 illustrates , in perspective view , the plenum facility comprises both passive (conduits, couplers, etc .) and active
shown in FIG . 1 ; (valves , pumps , sensors , etc . ) components ; a subset of the
FIG . 8 illustrates, in top plan view , the orifice plate passive components are shared , whereas, in general, the
portion of the plenum facility shown in FIG . 7 ; 10 active components are duplicated and adapted to cooperate
FIG . 9 illustrates , in perspective view , the chamber por- in operation as separate , redundant sub - facilities. Excluding
tion of the plenum facility shown in FIG . 7 ; the tank 14 , the primary shared component is the plenum
FIG . 10 illustrates, in top plan view , a plurality of appli facility 36 ( see , FIG . 1 and FIG . 7 ) comprising an orifice
ance slots distributed vertically along, and extending trans - plate 36a (see , FIG . 8 ) and a plenum chamber 36b (see , FIG .
verse to , a long axis of the tank of FIG . 1 ; 15 9 ). As can be seen in FIG . 1, cooled dielectric fluid is
FIG . 11 illustrates, in longitudinal cross -sectional view , pumped into both ends of the plenum facility 36 via a shared
the plurality of appliance slots distributed vertically along , distribution header 38 ( see , FIG . 2 and FIG . 3 ) . In general,
and extending transverse to , the long axis of the tank of FIG . the plenum plate 36a comprises at least one row of orifices
1: vertically aligned with each appliance slot 18a, with the
FIG . 12 illustrates, in flow schematic form , one instan - 20 dimensions and flow rates of each set being adapted to
tiation of a flow arrangement suitable for implementing our provide substantially equal flow of the dielectric fluid
invention ; and upwardly into each appliance slot 18a . Preferably, each
FIG . 13 illustrates, in control schematic form , one instan appliance slot 18a is supplied via several rows of orifices ,
tiation of a flow control facility suitable for implementing thus generally tending to reduce the volume of the dielectric
our invention . 25 fluid exiting each orifice and to make the flow of dielectric
In the drawings, similar elements will be similarly num - fluid more uniform upwardly through the appliance slots 18 .
bered whenever possible . However , this practice is simply One further shared component is the dielectric fluid recovery
for convenience of reference and to avoid unnecessary facility 40 (FIG . 2 ) comprising a dielectric fluid recovery
proliferation of numbers, and is not intended to imply or reservoir 42 (see , FIG . 3 , FIG . 4 and FIG . 13) positioned
suggest that our invention requires identity in either function 30 vertically beneath the overflow lip of the weir 22 and
or structure in the several embodiments . adapted smoothly to receive the dielectric fluid as it flows
over the weir 22; the dielectric fluid recovery reservoir 42 is
DETAILED DESCRIPTION OF THE further adapted to allow the recovered fluid to be removed
INVENTION from the reservoir 42 via redundant recovery ports 44a and
35 44b (only port 44a can be seen in FIG . 2 as the port 44b is
Shown in FIG . 1 (front view ) and FIG . 2 ( rear view ) is a obscured by the heat exchanger 32a ; but see FIG . 12 ). As
tank module 10 adapted for use in an appliance immersion can be seen in both FIG . 3 and FIG . 4 , we consider it
cooling system constructed in accordance with a preferred desirable to provide a vortex breaker at the input of each of
embodiment of our invention . For convenience of reference , the recovery ports 44 . Also , we provide a removable recov
we have illustrated in FIG . 1 the tank facility 12 of the 40 ery reservoir cover 46 adapted to also cover a major portion
immersion module 10 in partial cut-away to emphasize of the distribution header 38 ; note that, in both FIG . 2 and
several important internal facilities; we have shown the tank FIG . 3 , we have illustrated the reservoir cover 46 in a
facility 12 in isolation in FIG . 5 . In general, the tank facility partially raised orientation so as to better depict details that
12 comprises: a tank 14 adapted to immerse in a dielectric would otherwise be obscured. Note that we have constructed
fluid a plurality of electrical appliances 16 , e .g ., contempo - 45 the reservoir 42 such that the average height of dielectric
rary computer servers ( see, e . g., FIG . 11 ), each in a respec - fluid above the recovery ports 44 develops sufficient hydro
tive appliance slot 18a distributed vertically along , and static head to meet the requirements of the pumps 48 , while
extending transverse to , a long axis of the tank 14 ( see also tending to minimize the likelihood of breaking suction
generally, FIG . 10 ); an appliance rack facility 20 of con - during normal operation .
vention design adapted to suspend the appliances 16 (see , 50 At this point in the primary circulation facility 28, we
e . g ., FIG . 11 ) in respective appliance slots 18 ( see, FIG . 10 ); provide fully redundant sub - facilities 28a and 28b , each
a weir 22 ( best seen in isolation in FIG . 5 and FIG . 6 ), comprising a primary circulation pump ( 48a and 48b ) and
integrated horizontally into one long wall of the tank 14 associated passive and active components which , collec
adjacent all appliance slots 18 , and adapted to facilitate tively , provide the motive power for circulating the dielectric
substantially uniform recovery of the dielectric fluid flowing 55 fluid through the shared components and tank 14 . As can be
through each of the appliance slots 18 ; an interconnect panel generally seen , each of these sub -facilities 28a and 28b is
facility 24 attached to the upper rear edge of the tank 14 and adapted to recover the dielectric fluid exiting the tank 14 via
adapted to mount various appliance power distribution the weir 22 , re -pressurize the recovered fluid , pass the
equipment, cable interconnection panels and the like (none re -pressurized fluid through a respective one of the heat
shown); and a cover 26 adapted to be opened and closed 60 exchangers 32a and 32b , and then back to the plenum
from the front of the tank 14 (and which may include a facility 36 via the header 38 .
translucent portion to allow viewing of the interior of the Shown in FIG . 12 is one flow arrangement suitable for
tank 14 when in the closed position ). In addition to the tank integrating our tank module 10 into a fully redundant,
facility 12 , the immersion module 10 comprises: a primary appliance immersion cooling system , comprising the pri
circulation facility 28 (portions ofwhich are shown in both 65 mary circulation facility 28 and the secondary fluid circu
FIG . 1 and FIG . 2 ); a secondary fluid circulation facility 30 lation facility 30 . In general, the secondary fluid circulation
(of which only redundant heat exchangers 32a and 32b are facility 30 comprises redundant secondary circulation sub
US 10 ,405 ,457 B2
facilities 30a and 30b, each of which is adapted to circulate performed by the controllers 58 , 60 and 62 may be imple
a cooling fluid , e .g ., treated water, through the respective m ented in the form of dedicated application -specific soft
heat exchanger 32a and 32b to extract heat from dielectric ware executing on a conventional computer platform having
fluid counter-circulating therethrough and to dissipate to the the appropriate resources; indeed , it would be entirely fea
environment the heat so extracted . In the illustrated embodi- 5 sible to implement the entire control facility 56 on a server
ment, each of the secondary fluid sub - facilities 30a and 306
m 16 installed in a tank 14 .
comprise conventional cooling towers 50a (including fan One desirable enhancement that we recommend is a
facility 52a ) and 50b (including fan facility 52b ), and remote control facility , implemented , e. g ., via the master
secondary circulation pumps 54a and 54b . To facilitate controller 62 (or by way of a direct, per- controller interface ),
flexible operation in installations including multiple immer - 10 adapted to facilitate remote monitoring of system status
sion modules 10 in combination with a plurality of second - (e . g ., temperatures , pressures , etc .) and control over system
ary circulation sub - facilities 30 , a common header arrange control parameters ( e . g ., temperature and pressure limits ,
ment can be implemented as illustrated in the secondary etc .) to the primary controllers 58 and secondary controllers
fluid circulation loop , with flow control valves located at key 60. For example, using a conventional data communication
flow control points as is known. 15 hardware module 64 , e . g ., an ethernet card implementing the
Shown in FIG . 13 is a control facility 56 adapted to TCP/ IP protocol, a modern web browser can be adapted to
monitor and control the operation of both the immersion provide a graphical user interface (“GUI” ) with sufficient
module 10 (including all active components of the primary functionality to facilitate monitoring and control of an entire
circulation facility 28 ) , and the secondary fluid circulation installation from a remote location . Such a GUI may be
facility 30 . As will be evident to those skilled in this art, 20 implemented using any of a number of programming para
efficient operation of our immersion module 10 requires digms, e .g., PHP, .NET and the like .
continuousmonitoring and control of several essential oper - Operational control of redundant, continuous process flow
ating parameters , including fluidic temperatures, pressures , systems is generally well known . Preferable , each of the
conductivity and pH at several points in the primary and several redundant sub - facilities are routinely activated to
secondary circulation loops. Although the several sensory 25 assure current functionality , and to allow the inactive sub
and control functions can be implemented using traditional facility to be serviced according to an established schedule .
dedicated hardware components , we prefer to employ at We believe this continuous rotation of system resources to
least one programmable logic controller (“ PLC ” ), commer - be so important that we recommend switching the sub
cially available from any of a number of respected vendors , facilities at least once , and preferably, several times , per day ;
e . g., the Allen - Bradley brand of PLCs from Rockwell Auto - 30 although this is possible to implement manually, we prefer
mation , Inc. In the instantiation illustrated in FIG . 13, we to enable the master controller 62 to control the sequencing
have depicted : a primary controller 58a adapted to monitor of the several switch -over operations . One further aspect of
and control the operation of the primary circulation sub - this sophistication in control is the ability to perform stress
facility 28a as a function of the temperature of the dielectric testing of the several sub - systems under controlled condi
fluid in the tank 14 ; a secondary controller 60a adapted to 35 tions so as to assure appropriate response to real-time
monitor and control the operation of the secondary fluid emergencies.
circulation sub -facility 30a as a function of the temperature in our First Parent Provisional, we have disclosed an
of the dielectric fluid flowing through the heat exchanger alternate embodiment comprising an appliance immersion
32a ; and a master controller 62 adapted to coordinate the tank facility wherein the function of the plenum facility 36
activities of the primary controller 58a and secondary con - 40 is performed by a manifold facility comprising a ladder
troller 60a . As can be seen , we have incorporated into the arrangement of tubular spray bars, each bar of which sup
primary circulation sub - facility 28a: supply and return sen - plies dielectric fluid to a respective appliance slot. As we
sors , including a temperature probe, T, inserted into a noted , one particular advantage of this arrangement is that
thermowell (not shown ) installed in the bottom of the individual spray bars may be shut off if the respective
reservoir 42 adjacent a respective return port 44a ( note that, 45 appliance slot is not occupied and , thus, save energy. To
in FIG . 4 , only one of the holes that receive the thermowells further increase energy efficiency, we have provided
is illustrated ,but both holes are illustrated in FIG . 12 ); a pair optional vertical flow barriers adapted to partition the tank
of sensor facilities , S , which may sense temperature , pres - into an active portion , having active appliances, and a
sure and conductivity, as deemed desirable ); and return ( and, stagnant portion , having no active appliances . One further
if desired , supply ) flow control valves and controls for the 50 enhancement we disclosed is the provision of temperature
primary circulation pump 48a ; of course , a redundant set of sensors per appliance slot, such that the flow rate through
these components exists for the primary circulation sub - each spray bar can be dynamically varied as a function of the
facility 28b . In general, the goal is to maintain the tempera temperature of the dielectric fluid exiting the respective slot.
ture of the dielectric fluid in the tank 14 between a prede - Other operative configurations will be readily perceived by
termined minimum temperature and a predetermined 55 those skilled in this art.
maximum temperature . In a manner analogous to the embodiment described in
As noted above , we have provided separate control equip - our First Parent Provisional, it would be advantageous, from
ment cabinets 34a and 34b, each adapted to accommodate an energy point of view , to provide a plurality of flow barrier
the several components comprising a respective one of the plates 66 (shown by way of example only in FIG . 11 ), each
primary controllers 58a and 58b . For convenience of access, 60 adapted to be attached to the top of the plenum facility 36 so
we prefer to co -locate with each of the cooling towers 50 a as substantially to block the flow of the dielectric fluid
protective housing (not shown ) for the respective secondary through the row (s ) of orifices in the plenum plate 36a
controller 60 . Of course , the control facility 56 can be corresponding to at least a respective one of the appliance
instantiated as a single , multi -module PLC facility , with slots 18a ; an elastomeric layer ( not shown ) could be pro
similar or other combinations of monitoring devices as 65 vided on the interface surface of the plate ( s ) 66 to enhance
deemed most appropriate for a particular installation . Alter the sealing effect. Such an arrangement would allow the total
natively, one or more , and perhaps all, of the functions flow through the plenum facility 36 to be adjusted , in the
US 10 ,405,457 B2
field , as a function of the actual number of active appliances undesirable foreign matter that may have been picked up by
16 in the tank 14 . Further , this arrangement can incorporate the dielectric fluid on its passage though the entire primary
a relocatable vertical baffle plate 68 ( see FIG . 11 ) adapted circulation facility 28 ; chemical sensors may also be pro
substantially to partition the tank 14 into an active portion vided to detect the presence of unexpected chemicals that
14a containing the active appliances 16 and an inactive 5 may indicate failure of sub - components within one of the
portion 14b containing no appliances (or at least no active appliances 16 . Similar components, such as pH sensors ,may
appliances 16 ); preferably , the baffle plate 68 is adapted to also included in the secondary fluid circulation facility 30 .
be mounted in the appliance rack facility 28 in a manner As can be seen generally in FIG . 1 , we provide a pair of
similar to an actual appliance 16 (the baffle plate 68 need not low dielectric fluid level sensors 70a and 70b adapted to
fully block the flow of dielectric fluid between the active 10 trigger an alarm signal in the event that, for whatever reason ,
portion 14a and inactive portion 14b , but only significantly the level of the dielectric fluid in the tank 14 drops below a
impede the flow between these portions ). Note that, in the predetermined minimum level. Additionally, the responsive
example scenario illustrated in FIG . 11, we have shown one primary controller 58 can initiate other actions to address the
possible arrangement of a total of 8 active appliances 16 detected problem , including activating audio alarms, trans
distributed across 16 appliance slots 18a so as to spread the 15 mitting electronic alert signals and the like.
total heat load across adjacent empty slots 18a . Such an To solve a reciprocal problem , namely leakage from an
optimal arrangement is possible only if less than a majority external portion of the primary circulation loop 28 resulting
of the available appliance slots 18a are occupied by an active in the dielectric fluid in the tank 14 being back - siphoned
appliance 16 . Clearly , such optional adjunct facilities through the plenum facility 36 , we recommend incorporat
enhance flexibility in operation , accommodating dynamic 20 ing a siphon breaker 72 (see , FIG . 1 ) in the supply pipe at
adjustment of the flow rates in the primary circulation a predetermined location well above the plenum facility 36
sub- facilities 28a and 28b under variable heat loads, while but somewhat below the level of the weir 24 . Such a siphon
providing opportunities to conserve energy that might oth breaker can be as simple as a relatively small diameter hole
erwise be expended moving the dielectric fluid through the 72 drilled through the supply pipe at the selected location ;
inactive portion 14b of the tank 14 . Other operative con - 25 due to the relatively high viscosity of the dielectric fluid ,
figurations will be readily perceived by those skilled in this even when heated , any resulting leakage during normal
art. operation will be relatively insignificant. Other operative
In our Second Parent Provisional, we have disclosed responses to address these and other unusual fluidic condi
another embodiment comprising a more conventional, less - tions will be readily perceived by those skilled in this art.
modularized instantiation with appropriate flow and control 30 As is known ( see , e . g ., Best) , many conventional, com
facilities . In this embodiment, we chose to implement tank mercially available electrical/ electronic appliances include
clusters , comprising , e .g., 4 appliance immersion tank facili components that will not function correctly if immersed in
ties, with substantially all of the other equipment being a dielectric fluid , especially one as viscous as mineral oil:
constructed from stand -alone, commercially available com cooling fans and rotating media disk drives . In general, all
ponents . Such an arrangement offers greater opportunities to 35 cooling fans are unnecessary in an immersion cooling sys
select and install improved components, or to add enhance - tem and can be simply removed . Themedia drives , however,
ments to the installation , as deemed desirable after initial are usually necessary for normal appliance operation . One
installation . Other operative configurations will be readily option is to remove each drive , totally seal the drive against
perceived by those skilled in this art . fluid entry , and reinstall the now - sealed drive (pre -sealed
As we noted above with reference to the embodiment 40 drives are also available ). Another option is to remove the
illustrated in FIG . 12 , the secondary flow header facility is drive and mount it on the interconnect panel facility 24 ;
well adapted to allow any secondary circulation sub - facility typically special cabling will be required to re - attach the
30 to be connected to any active heat exchanger 32 . Such a drive to the internal appliance socket. Yet another option is
facility provides great flexibility in dealing with unusual to replace the rotating media drive with a solid - state drive
system conditions, especially in installations wherein the 45 having no moving components . Other operative configura
secondary circulation sub - facilities 30a and 30b are each tions will be readily perceived by those skilled in this art.
sized to support a cluster of tank modules 10 . Imagine, for It will be recognized that, in all of the embodiments
example , that, while one of the secondary circulation facili- described herein , emphasis was placed on minimizing the
ties 30 , say sub - facility 30a , is being serviced , the activities total volume of the dielectric fluid circulating throughout
of the set of appliances 16 in one tank 14 in the cluster are 50 each immersion module 10 . We submit that the key concept
higher than normal, resulting in a rise in temperature in that here is to move the secondary fluid to the point of heat
tank 14 above the desired maximum . In response , the master exchange with the primary fluid , rather than to move the
controller 62 can direct Primary Controllers 58a and 58b primary fluid to the point of heat exchange with the sec
assigned to tank 14 to operate both of the primary circulation ondary fluid . Thus , in our preferred embodiment, all of the
sub - facilities 28a and 28b simultaneously, i.e ., in parallel. 55 essential components of the primary circulation facility 28
Using the secondary flow header facility, the heat being are tightly co - located with the tank 14 so as to form a
extracted by both of the heat exchangers 32a and 32b may highly -integrated module . Further, our placement of the
be dissipated using the resources of the single on - line reservoir 42 outside of (but immediately adjacent to ) the
secondary circulation sub - facility 30b . Thus , one clear tank 14 tends to reduce the total volume of the dielectric
advantage of this alternate embodiment is the ability 60 fluid ( as opposed to the alternative arrangement we proposed
dynamically to perform load balancing across all system in our First Provisional, wherein a recovery trough was
resources . Other operative configurations to support sub - disposed within the tank 14 ); then , we positioned the com
system load balancing will be readily perceived by those ponents comprising the primary circulation sub - facilities 28
skilled in this art. so as to be vertically beneath the footprint of the reservoir
Preferably , one ormore filters (not shown ) are included in 65 42. In addition to conserving valuable floor space in a typical
the flow path through each of the primary circulation sub - data center installation , the resulting modular configuration
facilities 28a and 28b to remove any particulates or other facilitates both easy initial installation and subsequent
US 10 ,405 ,457 B2
10
upgrade to efficiently satisfy increasing data center work at least first and second primary circulation sub - facilities ,
loads. Indeed , our invention greatly enhances system scal each adapted to operate independently to circulate the
ability , a key concern to data center operators . Finally , our dielectric fluid through the tank ;
system -wide redundancy substantially assures fail -soft wherein the control facility is further adapted to coordinate
operation during periods of unusual environmental condi- 5 the operation of the first and second primary circulation
tions, infrastructure instability or political unrest. sub - facilities and the secondary fluid circulation facilities so
Although we have described our invention in the context to maintain the temperature of the dielectric fluid in the tank
of particular embodiments, one of ordinary skill in this art substantially between a predetermined minimum tempera
will readily realize that many modificationsmay be made in
such embodiments to adapt either to specific implementa - 10 ture and a predetermined maximum temperature .
5 . The system of claim 1 wherein the control facility
tions. By way of example, itwill take but little effort to adapt further comprises a communication facility adapted to facili
our invention for use with electronic appliances other than tate monitoring and control of the control facility from a
contemporary servers; and to adjust the dimensions of the remote location .
appliance accommodation slots accordingly . Similarly, prac 6 . A tank module adapted for use in an appliance immer
titioners in the art will readily recognize that other, known 15
secondary circulation facilities may be employed effectively , sion cooling system , the tank module comprising:
including forced air, vapor compression systems, earth a tank adapted to immerse in a dielectric fluid a plurality
water sink loops , waste heat recovery and recycling systems, of electrical appliances, each in a respective appliance
and the like (see , e .g ., the several alternatives discussed in slot distributed vertically along, and extending trans
Best ). Further, the several elements described above may be 20 verse to , a long wall of the tank , the tank comprising:
implemented using any of the various known manufacturing a weir , integrated horizontally into the long wall of the
methodologies , and , in general, be adapted so as to be tank adjacent all appliance slots , having an overflow
operable under either hardware or software control or some lip adapted to facilitate substantially uniform recov
combination thereof , as is known in this art. ery of the dielectric fluid flowing through each
Thus it is apparent that we have provided an improved 25 appliance slot; and ;
system and method of operation for immersion cooling of a dielectric fluid recovery reservoir positioned verti
appliances and the like . In particular, we submit that such a cally beneath the overflow lip of the weir and
method and apparatus provides performance generally com adapted to receive the dielectric fluid as it flows over
parable to the best prior art techniques butmore efficiently the weir ;
and effectively than known implementations of such prior 30 a primary circulation facility adapted to circulate the
art techniques. dielectric fluid through the tank , comprising :
What we claim is : a plenum , positioned adjacent the bottom of the tank ,
1. An appliance immersion cooling system comprising : adapted to dispense the dielectric fluid substantially
a tank adapted to immerse in a dielectric fluid a plurality uniformly upwardly through each appliance slot; and
of electrical appliances , each in a respective appliance 35 a control facility adapted to control the operation of the
slot distributed vertically along, and extending trans primary fluid circulation facility as a function of the
verse to , a long wall of the tank, the tank comprising: temperature of the dielectric fluid in the tank .
a weir, integrated horizontally into the long wall of the 7 . The module of claim 6 wherein the tank and primary
tank adjacent all appliance slots, having an overflow circulation facility comprise a highly - integrated module .
lip adapted to facilitate substantially uniform recov - 40 8 . The module of claim 6 wherein the tank further
ery of the dielectric fluid flowing through each comprises:
appliance slot; and ; an interconnect panel facility adapted to mount appliance
a dielectric fluid recovery reservoir positioned verti support equipment.
cally beneath the overflow lip of the weir and 9 . The module of claim 6 wherein the primary circulation
adapted to receive the dielectric fluid as it flows over 45 facility further comprises:
the weir ; at least first and second primary circulation sub -facilities ,
a primary circulation facility adapted to circulate the each adapted to operate independently to circulate the
dielectric fluid through the tank , comprising: dielectric fluid through the tank ;
a plenum , positioned adjacent the bottom of the tank , wherein the control facility is further adapted to coordinate
adapted to dispense the dielectric fluid substantially 50 the operation of the first and second primary circulation
uniformly upwardly through each appliance slot; sub - facilities so to maintain the temperature of the dielectric
a secondary fluid circulation facility adapted to extract fluid in the tank substantially between a predetermined
heat from the dielectric fluid circulating in the primary minimum temperature and a predetermined maximum tem
circulation facility, and to dissipate to the environment perature .
the heat so extracted ; and 55 10 . The module of claim 6 wherein the control facility
a control facility adapted to coordinate the operation of further comprises a communication facility adapted to facili
the primary and secondary fluid circulation facilities as
tate monitoring and control of the control facility from a
a function of the temperature of the dielectric fluid in
remote location .
the tank . 11 . A tank module ( 10 ) adapted for use in an appliance
2 . The system of claim 1 wherein the tank and primary 60 immersion cooling system , the tank module comprising :
circulation facility comprise a highly -integrated module . a tank (12 ) adapted to immerse in a dielectric fluid a
3 . The system of claim 1 wherein the tank further com plurality of electrical appliances ( 16 ), each in a respec
prises : tive appliance slot ( 18 ) distributed vertically along, and
an interconnect panel facility adapted to mount appliance extending transverse to , a long wall of the tank ( 10 ) , the
support equipment. 65 tank ( 10 ) comprising :
4 . The system of claim 1 wherein the primary circulation a weir 22 , integrated horizontally into the long wall of
facility further comprises : the tank ( 10 ) adjacent all appliance slots ( 18 ),
US 10 ,405 ,457 B2
11
adapted to facilitate substantially uniform recovery wherein the control facility is further adapted to coordinate
of the dielectric fluid flowing through each appliance the operation of the first and second primary circulation
slot (18 ); sub - facilities so to maintain the temperature of the dielectric
a primary circulation facility ( 28 ) adapted to circulate the fluid in the tank substantially between a predetermined
dielectric fluid through the tank ( 10 ) , comprising: 5 minimum temperature and a predetermined maximum tem
a plenum (36 ), positioned adjacent the bottom of the perature .
tank ( 10 ), adapted to dispense the dielectric fluid
substantially uniformly upwardly through each 14 . The module of claim 11 wherein the control facility
appliance slot ( 18 ); and further comprises a communication facility (62 , 64) adapted
a control facility (58 ) adapted to control the operation of to facilitate monitoring and control of the control facility
nction 10 from a remote location .
the primary fluid circulation facility (28 ) as a function
of the temperature of the dielectric fluid in the tank 15 . An appliance immersion cooling system comprising a
( 10 ). tank module according to any one of the preceding claims 11
12 . The tank module of claim 11 wherein the tank further through 14 .
comprises :
an interconnect panel facility (24 ) adapted to mount 135 16 . An appliance immersion cooling system according to
claim 15 , further comprising:
appliance support equipment. a secondary fluid circulation facility adapted to extract
13. The module of claim 11 wherein the primary circu heat from the dielectric fluid circulating in the primary
lation facility further comprises:
at least first and second primary circulation sub- facilities circulation facility , and to dissipate to the environment
( 28a, 28b ), each adapted to operate independently to 20 the heat so extracted .
circulate the dielectric fluid through the tank ;