The Datacenter Cheat Sheet: One size doesn't fit all

Choosing a datacenter is no trivial task. For the majority of sophisticated companies, the days of the one stop shop are history. This wasn't the case seven years ago when you weren't fired for choosing Exodus as your provider of datacenter services. It wasn't uncommon to add redundancy to your colo'd environment by installing gear in multiple Exodus facilities. Maybe one on the West coast and one on the East and you were a hero. Today Exodus is no more albeit some of their remnants remain with Savvis and Digital Realty Trust and a fresh crew of cautious datacenter providers have stepped up to the plate, each with a focus on fulfilling some market need. What could have possibly changed in the market to have such dramatic implications in such a short period of time? Networks(backbones, last mile options, overlay). Hardware got smaller but more dense. Software as a service is real. Ubiquity in access speeds and availability. Efficiency enablement...if there is such a phrase. Increase in demand for electicity. Access to capital. While all those don't need to be crossed off on everyones check list for most companies embarking on the discovery process of finding the right colo vendor, the bricks of the road for which we traveled.

The past two years have brought a change in the way datacenter vendors charge for the services they provide to customers. There was a time when you could compare the costs of doing business with one vendor to the other by their cost per rack or cabinet. If you're doing that today you could be in for a big surprise. In fact, if the first question your prospective vendor asks you is, 'how much space do you need?', you should consider making a bee line for the closest exit because the vendor should be telling you how much space you need based on your power requirement.

For most providers, a rack takes up 25 square feet of datacenter floor space. The issue is that floor space isn't all created equally. The 'players' today have varying power densities ranging from 60 watts a foot to 200 watts a foot. Thus, the rack in the 60 watts a foot facility provides less than 1/3 of the utility(in an economic sense) as the rack in the 200 watts a foot facility. I'm getting sidetracked but felt those were important points to make considering the newness of the industry in general and the lack of a standard unit of measure amongst vendors. I will dive deeper into the power issue and how to really compare one vendor vs another in my next post of power. And it may shock you. Sorry, couldn't resist :)

For simplicity, I will break down the buy side market and evaluate each section the way the supplier, not the buyer, would. We'll go generic and call the three sections:


Small(up to 100kw) - both in footprint(space) and power consumption(<3kw per rack)
- Customers in this category may have a requirement for a rack or two, perhaps up to ten. Power required for each rack is a single 20amp circuit and perhaps a redundant circuit as backup to the single 20amp. At 25 feet a rack and ten racks, this customer has a requirement of 250 sq ft and approx 30kw of power. This customer likely isn't as concerned with the cost of power as they are with the proximity of the datacenter to their office. The exception to that last statement would be with carriers and isp's who colo in carrier neutral sites as they tend to have relatively low power requirements and do most of their management via remote login.


Medium(between 100kw and 500kw) - in this category the vendor is most likely telling the customer how much space they'll need. Reason being, not all sites are equal. Example, a customer could go to Equinix whose pre 2007 sites are built to around 120 watts a foot and be told they need to purchase 125 cabinets worth of floor space in order to support their power requirement. In this example the customer may only need 50 cabinets with 10kw in each one. That is only two 208v 30amp power circuits per cabinet, or 100 power circuits total but the capacity on those circuits are 5kw each(actually 4.9kw at 80% of their gross capacity). This is a standard, run of the mill setup for companies like eBay, Amazon, Google(albeit different circuit types but roughly the same total kw per rack), Salesforce, Youtube, etc.


Large customers (greater than 500kw) - In addition to the customers mentioned above, this category includes most of the top 50 internet companies as well as many enterprises that may never colo their gear. The majority of users from this category are not colo'd in third party sites, they build and operate their own datacenters. While there are a handful that overlap(Yahoo, Facebook, Google, eBay, Internap, etc)those tend to be the internet related companies, not the traditional enterprise. I believe this is due to the nature of their growth and the fact that they had to start somewhere which likely was in a colo datacenter such as Exodus, Equinix, AT&T,e tc. Overtime, these types of companies built up their internal expertise in operating the physical components of a datacenter. This provided them them with the skill set to operate their own sites their growing demands for and expenses of outsourced datacenter space made investigating the possibility of building and operating their own sites a no brainer. Since running a datacenter is nothing new and not limited to internet companies, there is an established ecosystem of companies providing services and products to the operators of datacenters. As such, there are companies whose business it is to manage the day to day operations of any datacenter but the one with the asset on the book or the livelihood on the line, still have inhouse expertise in all the areas of design, operations, maintenance, etc. It's way too large of an investment and all too important to the day to day operations of the businesses in general to not have an inhouse staff of engineers and operations experts.

Along with the growth in demand for computing space came the revenue being generated from the services being delivered by the assets in these datacenters. More revenue = More access to capital + greater borrowing power + more flexibility in provisioning + less emphasis on planning + operational transparency to internal customers + increased costs controls + increasing costs of outsourcing = Decision to build, buy or lease and operate datacenter assets.

Before we get into who fits where, it may be useful to first go over why all vendors aren't going after the large customer and why if they do they limit the number of that type of customer to one per datacenter. Vendors want to fill their sites up and just like the buyers, they want to get the most bang for their buck. The difference is the vendors buck is already spent so they must get the most return on that buck. As with most markets, pricing is somewhat volume driven so the more you buy the cheaper it is. There really is no logic to that other than at some point the large customer would be spending enough if he was paying small customer pricing to put that spend towards building his own site and not outsourcing at all. For right or wrong that dynamic just is what it is.

At any rate, lets use an example using a ficticious vendor named XXX, inc. who just opened a 100k sq foot datacenter with 15MW of sellable power. XXX could fit 5000 cabinets in this space and deliver 150 watts a foot. XXX has a good sales team who uncovers two opportunities with prospective customers who have a requirement of 5MW each. In order to get these deals they will need to get real competitive on the pricing structure for the space for these customers, well below their retail rates. Often colo vendors are lured by landing the big name customer and the oomph such an incident would add to effort selling out a site quickly which pleases investors who are more likely to make additional if they can see success on the initial ones. Today, most of the big guys in the market have learned that giving all or most of the buildings resources to a single user are wise to the pitfalls wh customers is that they don't want to buy a bunch of additional services from along with such an arrangement.

The issue is that the you can't mix a large companys requirements with the colo vendors abilities and get a happy outcome because success to one may likely mean failure to another. It isn't so much that the low price hurts the vendors as it is the lack of operational flexibility the customers experience with vendors hurts the customers experience. In doing so, increasing the attractiveness of insourcing. For A perfect storm was in the brewing for a flood of investments being made in ensuring the ability to grow, and grow on your own terms, drive TCO down, less dependency on outside personnel and a bunch more factors, features and functions of running your own datacenter compared to colo'ing in a third party site.

This dynamic doesn't impact the vendors we know and love today as the opportunity cost of servicing a 5000 sq ft customer compared to selling that 5k sq ft to 200 different customers is far too great to not take notice. Assumptions for example: cab = 20sq ft, pricing excludes power, large deal would get 30% discount off retail, a cross connect costs $200 per month, big customers buy 10 cc's and small customers buy three.

Example of revenue generated by single customer in 5k sq ft:

250 cabs X ($560 discounted rate + $2000 for ten cross connects) = $142k
$142K divided by 5000 sq ft = $22.72 in revenue per sq ft


Example of selling the 250 cabs to individual customers:


250 cabs X ($800 per cab + 600 for cross connects)= $280K
$280k divided by 5000 sq ft = $56 in revenue per sq ft

Ahh, the reason it really isn't that good of a deal for a retail vendor to take too many anchor deals, the opportunity cost is huge!

The Data Center Cheat Sheet - What exactly are we dealing with?

It may be useful to go through a brief overview of Internet datacenter market history to properly appreciate todays market dynamics so bear with me if this is old news or a regurgitation of a not so happy time. Those times build character though :)


Over the past few years the Datacenter market has experienced a shift in power as it relates to the Datacenter Vendor and customer or prospective customer realtionships. This is a function of an imbalance in supply and demand. From 2000 to early 2005, it was a buyers market for colo and buyers played vendors off of each other to get the very best deals they could. And they were quite successful in getting the often desperate vendors to strike deals that were well below being financially healthy or sound. From the vendors perspective, they were just happy to get customers in their datcenters. After all, they had rent to pay to their landlords and sitting inventory that is not generating any money is worse than selling that inventory for anything greater than zero. Allot of poor pricing decisions were made during this window of time but they(pricing) weren't the only questionable attributes of the deals that went down during this period.

The bigger thorn in the side of most of these deals was related to what these customers were allowed to install in each of the racks or cages in the colo's. Remember the time and put yourself in a vendors shoes for a minute. You're negotiating with eBay or some other large retailer and just the thought of signing this customer makes you forget the notion of profitability. At this point stopping some of the bleeding will be a step in the right direction and as such you agree to give ebay the best rack pricing you've ever given anyone and don't put any parameters around how much power they can install and consume. Secretly you're really hoping they over provision power because that is money in your pocket that helps to offset the low rate on space you've agreed to. Sidebar definition: Over provisioning power is the scenario whereby a customer provisions 60 amps of primary 208v power(as an example) and only consumes 20amps of it. The customer pays the vendor for the full 60 amps but the vendor is only on the hook to the utility for what it uses, in this case 20amps. That is 40amps of profit right? Yes, at that particular month it was. This was quite a common situation and more often than not it was because the customers were ordering their colo configurations based on what their equipment required at full load. The issue here was that nobody was using the equipment to anywhere near capacity.

Slowly but surely the economy crawled back up and to the right(on a graphical basis) and with it came increased usage of the internet, ubiquity in broadband access, storage prices plummeting and innovation in usage of the internet in general. With the economy coming back more people were employed and they sure surfed the net at work(I think it would be intersting to see a study done on productivity output of employees with internet access and employees without it), more people had disposable income so they could afford the DSL or cable modem which allowed them to get further faster in their online worlds and gave them new ways to interact with one another via social networks which blended and intermixed with their real world lives. All of a sudden that steady 20amps of power consumption start to creep up. And up. And now frighteningly up. Up to the point that, as one VP of Ops of a big player in the space and who shall remain nameless, said,"this place could blow at any moment"

IMO, this was the point which the tables turned in favor of the vendors. By now the supply and demand was getting back to a state of equilibrium and it forced the datacenter vendors to do what I refer to as 'robbing Peter to pay Paul.' In order to fully grasp that notion you must understand what a datacenter really does. At the end of the day, a datacenter provides space, power and environmentals to it's customer sets. That is it. Datacenters don't provide managed services, service organizations do. Datacenters don't provide CDN or transit, ISPs and CDN's do. Datacenters don't provide storage, storage providers did that. We're talking about what the physical datacenter provides. Space, power, environmentals and physical security. Some may argue that these vendors provided interconnectivity and the vendors did but that was an added service layer that in actuality doesn't need to be a product of the vendor but could be the product of anyone or nobody(if it was free). When a datacenter is built you start out with a shell of a building and an amount of power that you can get delivered to that building. With that shell floor plan and that maximum amount of power available to you, you develop an overall layout of where things will go. Things being chiller, cooling towers, air handlers, generators, batteries, diesel storage, water storage, shipping and receiving, ingress/egress points, different authority levels of access, security and so on. You don't make these decisions without first knowing how much power you can get because there is a direct correlation to that amount of power and how many pieces of the Mechanical Electrical infrastructure plant will be required and how much square footage they'll occupy in the building. Long way of saying there is a finite amount of power and environental resources available for consumption. The standard increment or unit of measure in the market is either a rack or cabinet(42RU of actual space) or a sq ft. Each rack takes approx 20 sq feet of space on the datacenter floor. In order to forecast revenue, the datacenter operator simply takes the total sq footag of raised floor and divides by 20 sq ft to gt the # of available rack spaces they can sell, giving them some ability to forecast revenue. And they did forceast revenue based on these simplistic equality assuming assumptions. So if you have 50k sq ft of space you can sell 2500 cabinets. At $800/month per cabinet you'll generate $24MM in annual revenue. Sounds like a good plan right? The issue isn't it's simplicity but rather that it is only one piece of it, space. What about power? If you have 5Megawatts of power available for customer consumption across that 50k sq feet, you have a datacenter built to 100 watts a foot. If you have 7.5Megawatts of power available for consumption in that 50k sq ft, you have a datacenter built to 150 watts a foot. 10megawatts and you have 200 watts a foot. And so on.


Taking a step back, remember the example of the customer who was allowed to install the 60amps of 208v power in that single rack or those 20 sq ft? 60amps of 208v power in 20 sq ft equals about 500 watts a foot. Remember the notion of a finite amount of power coming in to the building and the linear relationship between power and the amount of space required for mechanical gear? That is because when delivering the power to the customers, the customers consume it with via the hardware infrastucture and in doing so, that hardware gets hot and gets hot quickly. Hence the beefy AC's that are required in datacenters. The same concept of the division of resources is carried over and applied to environmentals. We still don't have a global standard unit of measure for the industry because each building has different attributes and a customer may achieve higher utility in one vendors rack vs a different vendors rack because of the difference in the amount of available power in that rack. For this reason comparing Equinix rack pricing to Terremark rack pricing is useless unless you know the power per sq foot in each of their buildings. What point is there is trying to get Equinix who for examples sake has built out a datacenter at 200 watts a foot and is offering racks for $1000 each to lower it's rate to the $700 monthly fee that Terremark is offering in their datacenter which is built to 100 watts a foot. Don't you see what a screaming deal you already have with Equinix? To get that same functionality or utility at Terremark would cost you $1400 a month per rack.(Vendors and associated #s there are meant for expample purposes only). Circling back to the example earlier of eBay over provisioning those 60 amps of power or 500 watts a foot in the 100 watt per foot designed facility and you quickly realize that you, as the vendor gave up 5 racks of space and associated revenue for everyone one rack of space that eBay pays for. And pays for at the lowest rate you ever did. The deal is 5X worse than you thought. Not only that, but the perception of your company to a stranger walking in to your facility is that you are struggling because your datacenter is only 20% occupied spacewise because those first 500 racks that ebay installed consumed all of the power and cooling resources. Now imagine your the vendor who didn't catch this overprovisioning issue until you had oversubscribed your mechanical plant by a factor of 2 or 3X and you have all customers usage creeping up simultaneously. What do you do then? You say, "this place could blow at any moment" :) Those of us who lived through those types of situations and conditions will never get in them again. The first time around can be chalked up to ignorance. The second time would only be stupidity. This thought is evidenced by the hard lines the vendors take today as it relates to placing limits on the amount of power per rack they will allow their customers to install.

Taking the example from earlier with ebay using the entire pool of resources in 20% of the space of in the building and you can view it one of two ways. The first being that the supply of available space just shrank by 80% or the demand for space just increased by a factor of five. The market adjusted itself and the tables turned in favor the datacenter vendors and shows no signs that it will revert back to it's old ways. Sure, you hear allot about new datacenters being built today but remember, there hasn't been any signficant investment in this space in about ten years. During those ten years, computing clusters have gotten physically smaller and financially cheaper while increasing in performance. All of this resulting in more power consumption per rack unit, doing more in less space but with no change in the relative volume of an amp of power. Meaning the computers got more efficient in both performance and amount of space the physically take up but the power is what it is. And that is a study of physics. Efficiencies aren't a part of power, they're a part of those things that use power. Wrapping this up, the market has experienced all sorts of technological progress on hardware and software piece of the equation allowing users to pack more in to less but that less consumes exponentially more power than that more did in the previous scenario. The most scarce resource of a datacenter is power. And that means cooling too.

datacenters, data center, watts/ft, kw, kv, power, density, hvac, colo, equinix, savvis, terremark, internap, global crossing, exodus, amazon, salesforce.com, ebay, efficiency, amps, volts,