Advanced Data Centers, Inc. 50 Megawatt Sacramento Project

For those of you that have a blog or maintain a website you probably are familiar with web analytics. I was first introduced to web analytics in my days at Equinix where I managed the Omniture relationship. When I first began working with Omniture they had a couple racks of servers. By the time I left Equinix, Omniture was close to 1000 racks of equipment spread across the globe. This happened over the course of a couple years. It didn't take a dummy to figure out they were onto something. A couple months after I left Equinix I started blogging and was introduced to web analytics from the point of a customer/user. I believe analytics is one of the most powerful elements of online behavior. Since the internet is subsidized by advertising dollars analytics plays a huge role. But this post isn't about analytics, it's about, as the title suggests a new 50MW data center project underway in Sacramento, CA. 50 Megawatts is the most power at any data center site that I have ever come across. This project has been and will continue to be the recipient of a good portion of my attention for quite some time....or until we are out of space and power.

After spending 7 years at Equinix I thought I was done with data centers. So much so that it made my head spin even thinking about them. Shortly after leaving, it dawned on me that the data center market is unique in that there are natural barriers to entry in the form of extremely high cost of admission. Some people claim that data centers are being commoditized and with all the new construction taking place this line of thought is getting more attention. For those of us building data centers this is exactly what we want....that is of course as long as it's not the attitude of an investor or banker.

It's positive for us in that by virtue of it's negative connotation, it naturally scares new entrants away, creating additional strains on supply. It's also good because it makes it very difficult, if not impossible, for a new entrant to raise the necessary capital to build out a large scale data center due to construction costs of approximately $1500 per square foot. That cost is enough to scare most people away alone. There are quite a few instances of major corporations in the US forgoing the outsourcing model of colocation and becoming site operators themselves. Without huge augmentation, very few companies have the scale and resources to make this work over an extended period of time. Not impossible by any stretch but certainly challenging. Probably the most challenging element in getting total executive buy-in/support takes place when the CIO meets with the CFO, telling him that he needs $250MM to build a new data center that will support their IT requirements for the next ten years. More often than not, the CFO recalls a similar discussion 18 months ago in which this same CIO was asking for budget to outsource the same type of requirements, albeit less capacity, to a colo supplier. Once the CFO realizes this isn't a prank and that this guy is serious he collects himself and pours a stiff drink(it's a late night at the office in this example so its all good.)

When we embarked on our new venture, we had these types of discussions in mind and saw an opportunity to provide a service which would give the CIO the resources to support their IT customers and, at the same time, give the CFO a heart attack at every budget meeting.

I'm getting sidetracked so I'm going to get back to the analytics component. I find it very interesting to take a close look at how people get to this blog. Some get here by entering terms on the various search engines while some get here by clicking on links on other blogs or news sites that reference this one. And for some, I have no idea how they wound up here. It has been enlightening to see what people search for when they go to google, yahoo, or whatever search engine they use. I've found that a good portion of the folks who wind up here search for common terms that we throw around every day in the data center world...things like: watts per sq foot, kva to kw conversion, 60amp 208v circuit, Equinix rack price, Savvis data centers, 365main outage, etc. You get the point. The analytics service I use allows for me to see the address and domain of these readers network and has given me some insight to what IT people within these organizations have on their plate...or what keeps them up at night. Then again, I could be so wrong and off base I may as well be fire on an frozen pond. Either way, people who wind up here on my blog appear to be in the process of determining how to address their data center requirements.

Without overstepping the boundary of turning this blog into a sales pitch, I really believed the company behind the aforementioned data center project, Advanced Data Centers(ADC), may be a valuable partner to those folks in need of capacity for their IT gear. As such, what better way to reach them than to discuss it here.

If you're one of those folks or think you will be soon, I invite you to visit the ADC website to see what we're up to and if it looks intriguing drop me a note either on this blog or via the form on the contact page of the website.

The Data Center Cheat Sheet - What exactly are we dealing with?

It may be useful to go through a brief overview of Internet datacenter market history to properly appreciate todays market dynamics so bear with me if this is old news or a regurgitation of a not so happy time. Those times build character though :)


Over the past few years the Datacenter market has experienced a shift in power as it relates to the Datacenter Vendor and customer or prospective customer realtionships. This is a function of an imbalance in supply and demand. From 2000 to early 2005, it was a buyers market for colo and buyers played vendors off of each other to get the very best deals they could. And they were quite successful in getting the often desperate vendors to strike deals that were well below being financially healthy or sound. From the vendors perspective, they were just happy to get customers in their datcenters. After all, they had rent to pay to their landlords and sitting inventory that is not generating any money is worse than selling that inventory for anything greater than zero. Allot of poor pricing decisions were made during this window of time but they(pricing) weren't the only questionable attributes of the deals that went down during this period.

The bigger thorn in the side of most of these deals was related to what these customers were allowed to install in each of the racks or cages in the colo's. Remember the time and put yourself in a vendors shoes for a minute. You're negotiating with eBay or some other large retailer and just the thought of signing this customer makes you forget the notion of profitability. At this point stopping some of the bleeding will be a step in the right direction and as such you agree to give ebay the best rack pricing you've ever given anyone and don't put any parameters around how much power they can install and consume. Secretly you're really hoping they over provision power because that is money in your pocket that helps to offset the low rate on space you've agreed to. Sidebar definition: Over provisioning power is the scenario whereby a customer provisions 60 amps of primary 208v power(as an example) and only consumes 20amps of it. The customer pays the vendor for the full 60 amps but the vendor is only on the hook to the utility for what it uses, in this case 20amps. That is 40amps of profit right? Yes, at that particular month it was. This was quite a common situation and more often than not it was because the customers were ordering their colo configurations based on what their equipment required at full load. The issue here was that nobody was using the equipment to anywhere near capacity.

Slowly but surely the economy crawled back up and to the right(on a graphical basis) and with it came increased usage of the internet, ubiquity in broadband access, storage prices plummeting and innovation in usage of the internet in general. With the economy coming back more people were employed and they sure surfed the net at work(I think it would be intersting to see a study done on productivity output of employees with internet access and employees without it), more people had disposable income so they could afford the DSL or cable modem which allowed them to get further faster in their online worlds and gave them new ways to interact with one another via social networks which blended and intermixed with their real world lives. All of a sudden that steady 20amps of power consumption start to creep up. And up. And now frighteningly up. Up to the point that, as one VP of Ops of a big player in the space and who shall remain nameless, said,"this place could blow at any moment"

IMO, this was the point which the tables turned in favor of the vendors. By now the supply and demand was getting back to a state of equilibrium and it forced the datacenter vendors to do what I refer to as 'robbing Peter to pay Paul.' In order to fully grasp that notion you must understand what a datacenter really does. At the end of the day, a datacenter provides space, power and environmentals to it's customer sets. That is it. Datacenters don't provide managed services, service organizations do. Datacenters don't provide CDN or transit, ISPs and CDN's do. Datacenters don't provide storage, storage providers did that. We're talking about what the physical datacenter provides. Space, power, environmentals and physical security. Some may argue that these vendors provided interconnectivity and the vendors did but that was an added service layer that in actuality doesn't need to be a product of the vendor but could be the product of anyone or nobody(if it was free). When a datacenter is built you start out with a shell of a building and an amount of power that you can get delivered to that building. With that shell floor plan and that maximum amount of power available to you, you develop an overall layout of where things will go. Things being chiller, cooling towers, air handlers, generators, batteries, diesel storage, water storage, shipping and receiving, ingress/egress points, different authority levels of access, security and so on. You don't make these decisions without first knowing how much power you can get because there is a direct correlation to that amount of power and how many pieces of the Mechanical Electrical infrastructure plant will be required and how much square footage they'll occupy in the building. Long way of saying there is a finite amount of power and environental resources available for consumption. The standard increment or unit of measure in the market is either a rack or cabinet(42RU of actual space) or a sq ft. Each rack takes approx 20 sq feet of space on the datacenter floor. In order to forecast revenue, the datacenter operator simply takes the total sq footag of raised floor and divides by 20 sq ft to gt the # of available rack spaces they can sell, giving them some ability to forecast revenue. And they did forceast revenue based on these simplistic equality assuming assumptions. So if you have 50k sq ft of space you can sell 2500 cabinets. At $800/month per cabinet you'll generate $24MM in annual revenue. Sounds like a good plan right? The issue isn't it's simplicity but rather that it is only one piece of it, space. What about power? If you have 5Megawatts of power available for customer consumption across that 50k sq feet, you have a datacenter built to 100 watts a foot. If you have 7.5Megawatts of power available for consumption in that 50k sq ft, you have a datacenter built to 150 watts a foot. 10megawatts and you have 200 watts a foot. And so on.


Taking a step back, remember the example of the customer who was allowed to install the 60amps of 208v power in that single rack or those 20 sq ft? 60amps of 208v power in 20 sq ft equals about 500 watts a foot. Remember the notion of a finite amount of power coming in to the building and the linear relationship between power and the amount of space required for mechanical gear? That is because when delivering the power to the customers, the customers consume it with via the hardware infrastucture and in doing so, that hardware gets hot and gets hot quickly. Hence the beefy AC's that are required in datacenters. The same concept of the division of resources is carried over and applied to environmentals. We still don't have a global standard unit of measure for the industry because each building has different attributes and a customer may achieve higher utility in one vendors rack vs a different vendors rack because of the difference in the amount of available power in that rack. For this reason comparing Equinix rack pricing to Terremark rack pricing is useless unless you know the power per sq foot in each of their buildings. What point is there is trying to get Equinix who for examples sake has built out a datacenter at 200 watts a foot and is offering racks for $1000 each to lower it's rate to the $700 monthly fee that Terremark is offering in their datacenter which is built to 100 watts a foot. Don't you see what a screaming deal you already have with Equinix? To get that same functionality or utility at Terremark would cost you $1400 a month per rack.(Vendors and associated #s there are meant for expample purposes only). Circling back to the example earlier of eBay over provisioning those 60 amps of power or 500 watts a foot in the 100 watt per foot designed facility and you quickly realize that you, as the vendor gave up 5 racks of space and associated revenue for everyone one rack of space that eBay pays for. And pays for at the lowest rate you ever did. The deal is 5X worse than you thought. Not only that, but the perception of your company to a stranger walking in to your facility is that you are struggling because your datacenter is only 20% occupied spacewise because those first 500 racks that ebay installed consumed all of the power and cooling resources. Now imagine your the vendor who didn't catch this overprovisioning issue until you had oversubscribed your mechanical plant by a factor of 2 or 3X and you have all customers usage creeping up simultaneously. What do you do then? You say, "this place could blow at any moment" :) Those of us who lived through those types of situations and conditions will never get in them again. The first time around can be chalked up to ignorance. The second time would only be stupidity. This thought is evidenced by the hard lines the vendors take today as it relates to placing limits on the amount of power per rack they will allow their customers to install.

Taking the example from earlier with ebay using the entire pool of resources in 20% of the space of in the building and you can view it one of two ways. The first being that the supply of available space just shrank by 80% or the demand for space just increased by a factor of five. The market adjusted itself and the tables turned in favor the datacenter vendors and shows no signs that it will revert back to it's old ways. Sure, you hear allot about new datacenters being built today but remember, there hasn't been any signficant investment in this space in about ten years. During those ten years, computing clusters have gotten physically smaller and financially cheaper while increasing in performance. All of this resulting in more power consumption per rack unit, doing more in less space but with no change in the relative volume of an amp of power. Meaning the computers got more efficient in both performance and amount of space the physically take up but the power is what it is. And that is a study of physics. Efficiencies aren't a part of power, they're a part of those things that use power. Wrapping this up, the market has experienced all sorts of technological progress on hardware and software piece of the equation allowing users to pack more in to less but that less consumes exponentially more power than that more did in the previous scenario. The most scarce resource of a datacenter is power. And that means cooling too.

datacenters, data center, watts/ft, kw, kv, power, density, hvac, colo, equinix, savvis, terremark, internap, global crossing, exodus, amazon, salesforce.com, ebay, efficiency, amps, volts,

Sun's Blackbox

Get your mind out of the gutter, I'm referring to their portable datacenter. I was able to attend one of Sun's introductory briefings today in Menlo Park. When Jonathan Schwartz first announced this as a product I was very skeptical and threatened. Skeptical because these containers are 160 sq ft and can support a 200kw draw. That is 1250 watts per foot, albeit very isolated. And threatened because of the potential disruptive effect these new devices could have on the traditional datacenter market, my livelihood. Kinda.

I'm still skeptical but not as much as I was. I'm definitely not threatened, not because I don't believe in the viability but because the two are more complimentary than exclusive.

There are a few kinks to be worked out or how shall I say, items that are quickly set aside during their presentations but what did you expect? Marketing, marketing. Anyone know Al Hops?

Anyway back to this Blackbox.... A couple issues to note:

- these are NOT stand alone units. they require:


- multiple high voltage power connections in a minimum n+1 config - approx 250kw of provisioned primary power(these aren't connections you just run an extension cord f or. these are serious high voltage connections and as such require a serious infrastructure plant to get the connections down to the voltage required by the box. you don't call PG&E up and order one of these. typically this will be a branch on a larger power grid and in the datacenter world can be likened to a 12kv branch to a PDU.

- Cold water - Blackbox units require a cold water feed to support cooling off the payload, if you will. . To support 200kw of draw is approx 30 tons of chiller for these Blackboxes. The chiller doesn't come with the Blackbox and doesn't fit on or in one. Infact, a 60 ton chiller, enough capacity for 3 boxes, is about the size of a box itself. Chillers require power to produce cold water and you don't just plug a chiller into your wall outlet and be on your way. It requires the same or similar type of connections as the Blackbox, hi voltage, hi capacity circuits.

- Water Supply - HVAC systems will lose water to condensation, evaporation, leaks, overflows, etc and that water needs to be made back up to ensure smooth sailing. Maintaining N+1 design, you need two supplies of water from seperate suppliers. One is obviously your regular water supply but what about the second? dig a well like most datacenters do?

- UPS systems. There aren't any. Seriously. So that should tell me who the target customer is. Someone who doesn't care about uptime? The why the hell buy all this crap? why not host it on Amazon S3 or MediaTemple? Who doesn't care about uptime? Google is the only company I can think of, actually amazon too, who wouldn't care if they lost 8 racks of servers. I just don't think Sun is far enough along to have a solution for UPS that doesn't make you take a step back and say, 'wait a second, where the hell am i going to park five* tractor trailers so i can operate my 24 racks?' * 3 actual Blackbox container, 1 container for Generator and batteries and one container for the chiller.

I sound like I'm bagging on Sun but I'm not really. I like the idea and know it's a definite winner in niche applications such as military use, natural disaster use, isolated locations where it can be airlifted in and so on.

The thing is, if Sun owned the entire market for those specific applications it still isn't going to get Sun where it needs to be, it's just too limited in size. Sun needs to find a way to make these Boxes the defacto standard choice when a company begins evaluating datacenter options. That or sell the concept to the colo vendors by delivering them value by showing that the Boxes can compete economically with a standard raised floor environment. Coincidentally, just like a regular datacenter, in order to support a few of these boxes you will need a significant MEP resource which is essentially the bread and butter of a datacenter and datacenter operators are experts are managing MEP. Its a nice fit.

I liken the potential of Blackbox type architecture to what consumers are using Amazon S3 grid or google's own infrastructure(googleOS) for, a shared IT resource that supports unique data for each user and leverages commonalities among users. everything is virtually connected and resources are shared so if one goes down it doesnt matter yet the performance benefits of close proximity is omnipresent.

Cost. The fully built out container(without the computers, chiller, generator and truck or helicopter to transport it) currently costs $500k to build. Sun eluded to the price point of $250k as one which they're shooting for. $250k for 200kw isn't a bad deal. Equinix spends about $25k per rack or $1000/sq ft for a 2.5kw rack. In gross #'s Suns Box looks good at $1200/kw on the Box while a traditional datacenter, per Equinix's rough costs, comes in at $10,000 per kw. I don't know what the cost of the chiller plant and elctrical switches, etc would be but imagine it can't be more than 60% of the total costs of construction of the traditional so add another $6000 per kw and mutiply that sum, $7200, by the number of kw draw and you get your total cost for the Box and the supporting MEP gear. In this case it is $1.4MM for 200kw of datacenter equivalent. For Equinix, it would cost $2MM+

Lots of potential with this product but in order to be mass adopted it needs to demonstrate an economic benefit in addition to the obvious operational ones.