Server room rack power
 

Ok, not strictly D-I-Y i know, but I have been asked to come up with
some specs so that we can get quotes for a new power supply for our
server room at work.

W have 3 tall racks (30U from memory) round about 6 - 7 feet anyway
which are mostly stuffed full of Dell servers. These are rated at 2
Amp (The 1U jobbies) and 5A (2U jobbies). I don't know if this is the
maximum they draw at start up, or if they may take more at startup.
What I do know is that the 30A breaker which feeds the radial circuit
they all go back to (via a few UPSs) trips out if the power comes back
on after an outage

This can be a bit embaressing if we are all asleep in our beds and
there is a power cut. The sequence goes something like

Power goes off,

UPSs continue to power equipment

Backup generator kicks in

MCB opens

UPSs keep the servers up for 10 or 15 mins

Then everything goes down.

If we are here when it happens, we just put the breaker back in while
the UPSs are holding up and live to fight another day.

Anyway, to the question (I hope I haven't asked this before, I
couldn't find it via google) Is there a rule of thumb for specing
power supplies for these racks? Based on the rating plates, I could
allow 2.5A per U, but I don't know if I need to allow extra over what
it says on the rating plates of the servers, or if Dell have already
done that. Does 60A per rack sound about right, way over the top?
Way under?

Assuming it is right, what would be the best way to provide it? A
number of 16A or 30A Industrial sockets?

Answers on a postcard please. :+))

Server room rack power
 

Server room rack power
 

Andy Burns coughed up some electrons that declared:

I'd be tempted to split the load across multiple 16A outlets, to reduce
the total inrush current per circuit, also switch to type-C MCBs.

Agree. At Imperial College, we had this problem - had to go to type-D
breakers in some cases, but the design considerations for putting type D in
are onerous (eg you need a *very* low L-E impedance back to the supply:
fortunately the 11kV transformers were round the back of the building and
the College has a resident Electrical Engineer to worry about these
things).


Alternatively consider intelligent power distribution strips which can
sequence the servers back on after any power cut, and have the advantage
you can remotely power them off/on if required.

APC masterswitches were good at this, and allowed for remote power switching
via RS232 and ethernet. Think that range has been superceded, but it's
likely APC still make something similar.

Cheers

Tim

Server room rack power
 

Tim S wrote:
Andy Burns coughed up some electrons that declared:

I'd be tempted to split the load across multiple 16A outlets, to
reduce the total inrush current per circuit, also switch to type-C
MCBs.

Agree. At Imperial College, we had this problem - had to go to type-D
breakers in some cases, but the design considerations for putting
type D in are onerous (eg you need a *very* low L-E impedance back to
the supply: fortunately the 11kV transformers were round the back of
the building and the College has a resident Electrical Engineer to
worry about these things).


Alternatively consider intelligent power distribution strips which
can sequence the servers back on after any power cut, and have the
advantage you can remotely power them off/on if required.

APC masterswitches were good at this, and allowed for remote power
switching via RS232 and ethernet. Think that range has been
superceded, but it's likely APC still make something similar.

Cheers

Tim

I'm assuming the ups's keep the pc's running and the breaker supplies the
ups's ?

Therefore isn't it more likely the ups's overloading the circuit when the
power is returned ? they can have quite hefty startup currents, especially
if the batteries are already under load and discharging rapidly.

Pete

Server room rack power
 

Pete Cross wrote:
Tim S wrote:
Andy Burns coughed up some electrons that declared:

I'd be tempted to split the load across multiple 16A outlets, to
reduce the total inrush current per circuit, also switch to type-C
MCBs.
Agree. At Imperial College, we had this problem - had to go to type-D
breakers in some cases, but the design considerations for putting
type D in are onerous (eg you need a *very* low L-E impedance back to
the supply: fortunately the 11kV transformers were round the back of
the building and the College has a resident Electrical Engineer to
worry about these things).

Alternatively consider intelligent power distribution strips which
can sequence the servers back on after any power cut, and have the
advantage you can remotely power them off/on if required.
APC masterswitches were good at this, and allowed for remote power
switching via RS232 and ethernet. Think that range has been
superceded, but it's likely APC still make something similar.

Cheers

Tim

I'm assuming the ups's keep the pc's running and the breaker supplies the
ups's ?

Therefore isn't it more likely the ups's overloading the circuit when the
power is returned ? they can have quite hefty startup currents, especially
if the batteries are already under load and discharging rapidly.

Usually in a rack setup like this each server has two PSUs, one supplied
direct from mains (with generator backup) the other supplied from mains
via UPS.

There is no initial disruption to the half of the PSUs being fed from
the UPS, so no surge, when the mains goes off there will be a surge when
half the PSUs come back on line when either the generator kicks in, or
the mains is restored.

If this surge trips the MCBs then you leave the servers on one PSU
permanently, if this UPS happens to be on the same MCB they'll all die
when the battery is exhausted.

Server room rack power
 

I see now then...

So what you need is a delayed start on the psu's, same as on a sunbed where
the tubes are powered sequentially otherwise it would exceed the 13A fuse in
the plug.

Pete

Server room rack power
 

On Thu, 08 Jan 2009 11:12:25 +0000, Andy Burns wrote:

I'd be tempted to split the load across multiple 16A outlets, to reduce
the total inrush current per circuit, also switch to type-C MCBs.

Alternatively consider intelligent power distribution strips which can
sequence the servers back on after any power cut, and have the advantage
you can remotely power them off/on if required.

Both, I'm surprised racks with so many servers don't already have master
switches just for the sequential start up, the abilty to power cycle
individual bits of kit without having to physically visit is just a useful
side effect. B-)

As for power, per rack 60A is 13.8kW, 3 racks 41.4kW I hope you have some
good air con! Maybe buying a decent clamp meter and measuring what is
actually being drawn would be useful. Not sure how they are affected by
distorted waveforms produced by loads a SMPSUs though...

--
Cheers
Dave.

Server room rack power
 

Answers on a postcard please. :+))

It sounds like a classic case of your employer wanting to do it on the cheap
the result of which could be that you get the blame when it all goes pear
shaped.

The whole job needs to be specified by a specialist professional in the
field, which is not likely to be you average sparky who, with all due
respect to inhabitants of this NG, will no experience in this sort of
installation. If possible get three different quotes for the specification
and the installation of it. Get references for similar work they have done
and take them up.

Peter Crosland.

Server room rack power
 

Dave Liquorice wrote:

I'm surprised racks with so many servers don't already have master
switches just for the sequential start up,

You can get "dumb" sequential startup strips too

the abilty to power cycle
individual bits of kit without having to physically visit is just a useful
side effect.

Which you may not need if your servers have "lightsout" control where
you can turn them on and off over a dedicated network which is powered
from the standby power of the PSU.

Also with servers on multiple power supplies you need to have multiple
master switches and power off all relevant outlets feeding a single
server to reboot it.

Server room rack power
 

In article ,
wrote:
Ok, not strictly D-I-Y i know, but I have been asked to come up with
some specs so that we can get quotes for a new power supply for our
server room at work.

W have 3 tall racks (30U from memory) round about 6 - 7 feet anyway
which are mostly stuffed full of Dell servers. These are rated at 2
Amp (The 1U jobbies) and 5A (2U jobbies). I don't know if this is the
maximum they draw at start up, or if they may take more at startup.
What I do know is that the 30A breaker which feeds the radial circuit
they all go back to (via a few UPSs) trips out if the power comes back
on after an outage

We have 50 odd racks in our machine room - and often had this problem.

The Dell servers were by far the worst culprits along with the EVA storage.
Sun and HP servers tend to be better. We have dual psu machine feed from
two independant distribution networks and most Dell boxes appeared to
run from one or other psu on not share the load between them. This tended to
cause one side of the rack to be overloaded and cause the breakers to pop :-(

Sorted by fitting different MCBs in the SAN racks and decent sequencing
power distribution to the server racks. We use APC power dist (I think
the models we have are http://www.apc.com/products/family/index.cfm?id=136 )
which give us remote (ssh or https) control of the outlets and also
sequencing (can be configured with timings to allow things to boot in
order). They also offer some load monitoring.

The ones we have are zero U models that fit nicely into our APC racks - you
might be better with a few of the rack mount versions if you don't have
room to mount them.

Not cheap, but pretty solid

Darren (oh, we stopped buying Dell servers as well :))

Server room rack power
 

In article ,
Andy Burns wrote:

Usually in a rack setup like this each server has two PSUs, one supplied
direct from mains (with generator backup) the other supplied from mains
via UPS.

Or even better, both feeds from UPS protected feeds, with gen backup.

Of course, then someone buys a bloody sun server with one power lead or
even worse, one with *3* power feeds.

Stupid bloody machines. Thankfully, I think sun have stopped that now :)

Darren

Server room rack power
 

On Fri, 09 Jan 2009 09:04:07 +0000, dmc wrote:

In article
,
wrote:
Ok, not strictly D-I-Y i know, but I have been asked to come up with
some specs so that we can get quotes for a new power supply for our
server room at work.

W have 3 tall racks (30U from memory) round about 6 - 7 feet anyway
which are mostly stuffed full of Dell servers. These are rated at 2 Amp
(The 1U jobbies) and 5A (2U jobbies). I don't know if this is the
maximum they draw at start up, or if they may take more at startup. What
I do know is that the 30A breaker which feeds the radial circuit they
all go back to (via a few UPSs) trips out if the power comes back on
after an outage

We have 50 odd racks in our machine room - and often had this problem.

The Dell servers were by far the worst culprits along with the EVA
storage. Sun and HP servers tend to be better. We have dual psu machine
feed from two independant distribution networks and most Dell boxes
appeared to run from one or other psu on not share the load between
them. This tended to cause one side of the rack to be overloaded and
cause the breakers to pop :-(

Sorted by fitting different MCBs in the SAN racks and decent sequencing
power distribution to the server racks. We use APC power dist (I think
the models we have are
http://www.apc.com/products/family/index.cfm?id=136 ) which give us
remote (ssh or https) control of the outlets and also sequencing (can be
configured with timings to allow things to boot in order). They also
offer some load monitoring.

The ones we have are zero U models that fit nicely into our APC racks -
you might be better with a few of the rack mount versions if you don't
have room to mount them.

Not cheap, but pretty solid

Darren (oh, we stopped buying Dell servers as well :))


BAe Systems at Warton had a lot of problems with circuit breakers feeding
computer suites. They eventually solved it by going back to fuses! We
supplied them with some rather nice (if I may say it myself) fused
distribution panels.

--
Mick (Working in a M$-free zone!)
Web: http://www.nascom.info
Filtering everything posted from googlegroups to kill spam.

Server room rack power
 

In message , dmc writes
Darren (oh, we stopped buying Dell servers as well :))
Good man. PERC controllers are the spawn of Satan.



--
Clint Sharp

Server room rack power
 

wrote:
Ok, not strictly D-I-Y i know, but I have been asked to come up with
some specs so that we can get quotes for a new power supply for our
server room at work.

W have 3 tall racks (30U from memory) round about 6 - 7 feet anyway
which are mostly stuffed full of Dell servers. These are rated at 2
Amp (The 1U jobbies) and 5A (2U jobbies). I don't know if this is the
maximum they draw at start up, or if they may take more at startup.
What I do know is that the 30A breaker which feeds the radial circuit
they all go back to (via a few UPSs) trips out if the power comes back
on after an outage

This can be a bit embaressing if we are all asleep in our beds and
there is a power cut. The sequence goes something like

Power goes off,

UPSs continue to power equipment

Backup generator kicks in

MCB opens

UPSs keep the servers up for 10 or 15 mins

Then everything goes down.

If we are here when it happens, we just put the breaker back in while
the UPSs are holding up and live to fight another day.

Anyway, to the question (I hope I haven't asked this before, I
couldn't find it via google) Is there a rule of thumb for specing
power supplies for these racks? Based on the rating plates, I could
allow 2.5A per U, but I don't know if I need to allow extra over what
it says on the rating plates of the servers, or if Dell have already
done that. Does 60A per rack sound about right, way over the top?
Way under?

Assuming it is right, what would be the best way to provide it? A
number of 16A or 30A Industrial sockets?

Answers on a postcard please. :+))


Try replaceing the power distribuiton units with
http://www.olson.co.uk/sequential_19.htm so not everything turns back
on at the same time.

I do also hope you have high integrity earthing as well.

Server room rack power
 

In article ,
(dmc) writes:
The Dell servers were by far the worst culprits along with the EVA storage.
Sun and HP servers tend to be better. We have dual psu machine feed from
two independant distribution networks and most Dell boxes appeared to
run from one or other psu on not share the load between them. This tended to
cause one side of the rack to be overloaded and cause the breakers to pop :-(

Well, the whole point of splitting the power is so that if one
supply fails, the other continues to run the servers. If either
supply isn't capable of doing that, then there's no point in
splitting the servers across multiple supplies.

--
Andrew Gabriel
[email address is not usable -- followup in the newsgroup]

Server room rack power
 

In article ,
Andrew Gabriel wrote:
In article ,
(dmc) writes:
The Dell servers were by far the worst culprits along with the EVA storage.
Sun and HP servers tend to be better. We have dual psu machine feed from
two independant distribution networks and most Dell boxes appeared to
run from one or other psu on not share the load between them. This tended to
cause one side of the rack to be overloaded and cause the breakers to pop :-(

Well, the whole point of splitting the power is so that if one
supply fails, the other continues to run the servers. If either
supply isn't capable of doing that, then there's no point in
splitting the servers across multiple supplies.


Yes, but the start up surge on the dual psu dells was all on one psu. Given we had
plenty of them it meant seriously under speccing the power feeds and hoping they
wouldn't all come up and suck power from the same feed. It was fairly random IIRC
(at least if it was consistant which psu they would pick we could have shared
them about).

Sods law meant that 80% of them would come up on one of the feed tripping it out
are which point they would all switch to the other feed and the surge would trip
that.

As I say, we switched to HP for winblows servers and it got a lot better (maybe
just better psus?) and now we have all the racks sequenced anyway so the problem
has gone for good (he says, touching wood/mdf :))

Darren

Server room rack power
 

Clint Sharp wrote:
Good man. PERC controllers are the spawn of Satan.

Don't suppose you'd care to expand on that? All my critical systems are
on some sort of PERC...

Andy

Server room rack power
 

In message , Andy Champ
writes
Clint Sharp wrote:
Good man. PERC controllers are the spawn of Satan.

Don't suppose you'd care to expand on that?
Not really, apart from *never* listen to Dell support if they tell you
to re-seat a failing/failed disk, demand an exchange or replacement.

All my critical systems are on some sort of PERC...
Ahh. Of course you test your backups regularly I hope.

In fairness the newer PERC controllers do seem to be a lot better so far
but....

Andy

--
Clint Sharp

Server room rack power
 

On 11 Jan, 00:05, (Andrew Gabriel) wrote:

Good stuff as he usually does.

Many thanks one and all, and apologies for not being around for a
couple of days to read the replies. It has given me lots of food for
thought.

Server room rack power
 

Clint Sharp wrote:
Not really, apart from *never* listen to Dell support if they tell you
to re-seat a failing/failed disk, demand an exchange or replacement.

People fall for that? When that happens to me I regard it as good luck
that I've a warning of an upcoming failure!

All my critical systems are on some sort of PERC...
Ahh. Of course you test your backups regularly I hope.

IT department look after that. TBH I trust at least one of them.

Andy

Server room rack power
 

In message , Andy Champ
writes
Clint Sharp wrote:
Not really, apart from *never* listen to Dell support if they tell
you to re-seat a failing/failed disk, demand an exchange or
replacement.

People fall for that? When that happens to me I regard it as good luck
that I've a warning of an upcoming failure!
Far too regularly, despite having been warned several times, otherwise
intelligent people in jobs that have much higher pay rates than mine do
this and then call me out to mop up the damage.

Plenty of other PERC disasters as well, but that's the most common.

All my critical systems are on some sort of PERC...
Ahh. Of course you test your backups regularly I hope.

IT department look after that. TBH I trust at least one of them.
Lol, good, but do your IT department actually test the backups by
restoring to different hardware/disk space or do they just rely on
verifying it?

Andy

--
Clint Sharp