Server room design

PurpleIT

So my company is looking at moving and I need to come up with some rough estimates for what the new server room will cost. I’ve never built one from the ground up before and I don’t want to base much off of our existing room because it is so over-sized and poorly designed.


This won’t need to be too big; there are currently 3 four-post server racks in place and the equipment that is spread over a dozen two-post racks could be consolidated into 4 or 5. I need to allow for about 50% growth, but I would still consider this not a very large room. We aren't going to be going crazy trying to optimize the airflow, etc – we just need a no-frills server room that will have some room for expansion.

Does anyone have a book or a website that goes over standard spacing between racks, allowable space for loading equipment, price calculators, etc?


Any tips from people that have gone through this in the past? What do you wish you had done and what turned out to be a waste of time and money?

Any tips about power, cooling, layout, etc are welcome as well.

WafflesAndRootbeer

There are companies that do what you've been asked to do so you really need to touch base with them and look at the new building you are moving into. A lot of buildings aren't set up for "server rooms" so you may need to work with your employer to find a suitable building that is already equipped or designed to handle a small data center setup. When I worked for a banking company, their server rooms were repurposed office space or closets with no real ventilation or telecom infrastructure and you don't want to get stuck with that!

joehalford01

Only advice I can give is:

Make sure you have dedicated cooling.
You can control access - not shared with any other departments.
Not in a basement.

APC has guides online geared toward using them for heating/cooling and layout but it might help you.

DevilWAH

Joe mentions a good point, on the APC website you can build up you server and coms racks and it will give you an estimated power usage / UPS capacity you need to support it and heat output. Really good place to start to get an idea of the requirements you will need to consider.

For Power a decent UPS is a must, either individual rack UPS's or a room based system, If possible you want to have two for redundancy with each rack powered via both. Here it is worth getting a qualified electrician and some consultancy as there are a number of ways to do it. If you have a three phase supply you need to be careful as devices with multiply power supplies should be powered from a single phase.

once you have completed the UPS tool, it will also tell you estimated KW's the equipment will draw, this can easily by converted to BTU units which are a unit of thermal output as in a server room almost all energy used is converted in to heat that you need to remove.

Kilowatts (kW) to BTU/hr conversion calculator

once you have this value you can then look at your cooling solution to make sure it will be effective, I suggest non water based system and not mounted directly above the server racks, use ducting if you want a direct air flow. I have seen cooling units placed above racks get a leak and its not a good effect. For a small server room wall mounted units work well and keep the floor space free. The ideal solution is a split system with the cooling unit mounted outside the building and running the refrigerant into the server room, this again free's up space.

Lastly decided do you want to keep all the network in separate racks and install patch panels in the server racks running back to coms racks, or have Top or Rack switches in each server rack and trunking the network back to your core? I personal like the TOR switches, where I pair up racks so servers are patched both in to there own TOR switches and the next door rack to offer more resilience.

A lot of it depends on how much money and what you want to achieve.

First make sure you have enough power/cooling/space and connectivity as if you don't get this right you wont have a working server room. Then its a case of how much expansion and resilience you want to pay for. I always start with two solutions, the minimum that will work, and the money no object perfect dream. I then look for the sweet spot between the two that balances cost and functionality.

Trifidw

DevilWAH wrote: »

If you have a three phase supply you need to be careful as devices with multiply power supplies should be powered from a single phase.

Multiple inputs on a single power supply should be the same phase.
Different power supplies for the same server can be on different phases. DC has no concept of phase.

santaowns

What type of company is this ie big small medium? How many servers? What kind of networking is needed? How big of a room is needed? What does the budget look like?

DevilWAH

Trifidw wrote: »

Multiple inputs on a single power supply should be the same phase.
Different power supplies for the same server can be on different phases. DC has no concept of phase.

Indeed but I prefer to insure that each rack is run from a single phase to avoid any confusion, and just tell every one not to mix phases on a single box.

Trifidw

What if you have an issue with the UPS on one of the phases? We have a phase per side of the rack so the left PSU is always fed from the left side phase and the right PSU on the right phase (and the phases to each rack alternated as there are 3 of them).

santaowns

Here are some good ideas, Hot and cold aisles are a must, make sure that any space that is at the front of the rack is sealed with either filler panels or something else so that all air is going through the servers themselves, power each device with two sources or 2 pdu/ups setups so that if one fails you can still survive while it is being repaired, make sure each device has a kvm and that a crash cart is in each room or on each floor etc, Wiring should be redone everytime a server is replace to ensure that the cables work (recycle old cables and use that for dinners or something for the teams), extra parts on site are an awesome way to help alleviate some downtime(but is not essential if you have a 4x24x365 contract). I am sure there is a lot more that can be said but I still am waiting for the above questions to be answered.

Priston

If this is critical gear then you should definitely get redundant cooling units, what's your current power situation are you using L21-30? 3 phase 208? single phase? how many amps or KW does everything use?

Do you know the dimensions of the room your moving into or do you get to pick?

PurpleIT

Wow, it looks like I've got some catching up to do here...

We are moving into an empty shell so there is a lot of flexibility, but because the existing room (which will be used as a reference for all the non-technical people) is essentially a converted conference room it will be tricky to convince the powers that be that some things such as redundant cooling are needed. We have gotten by with much less for many years, so they are likely to doubt that we need it now.

We currently have a roof mounted AC unit (one of 15 or 20 total for the building) that is dedicated to the room, but it tends to go out 1-3 times a year and while the room gets up to 90+ degrees at these times, we haven't lost any equipment due to heat. We have been able to get by with propping open the door and using a couple of box fans.

The are talking about putting the new room in the "Garden Level" (i.e. basement) which I strongly oppose, but we will see if politics trumps common sense.

Power:

Power is currently handled by in-rack UPSs with a backup generator that will power the room along with emergency lights in the building. They have stated they want a generator at the new place so the we will probably stay with the in-rack units. Although we have not had 100% success with the generator the majority of power events are short enough that the UPSs get us through. In two cases over the last 11 years we have had a power loss and generator failure that caused everything to shut down.

I typically power servers or other devices with multiple power supplies by running each power supply off of separate UPSs. When I am able I put the UPSs on separate phases. They are always on separate circuits.

Network:

This is really a complicated area.

I'm fan of ToR switches, but there will need to be some patch panels with runs to other parts of the room simply because there will be multiple networks that MUST be physically isolated. This makes the networking portion very tricky as some people may have two or three computers plus their phone. Because of the way people move around all desks will need to be able connect to any combination of networks. None of this is too tricky until you try to figure out the locations of the patch panels in relation to switches and multiple comm racks. If one network only has 15 users its hard to justify multiple switches in multiple racks.

I think we are looking at 6 drops to each desk/cubicle and I can envision situations where 3-5 of those would be in use. I will probably make the phone and Network 1 the most integrated (that is to say the switches and patch panels are closest to each other) while the other networks will have switches in adjacent racks.

I will look at the APC site again, I saw mention of their various tools, but didn't actually find the links to download any applications. I will almost certainly look at working with a local company when it comes time get the detailed specs and do the build, but for now I am just looking to make sure I am not missing anything important and can work towards giving my boss something better then a wild guess as far as cost.

Thanks for the input; keep it coming!