Level3’s smoke and mirrors

Level3 didn't slash pricing for CDN service with their recent 'same price as transit' marketing ploy. If anyone slashed pricing, Amazon did with the introduction of their S3 storage. All Level3 did was imply their baseline CDN service is no better than their transit....heck, at least they admit it. A couple points to note. Baseline CDN service really is transit or no better than transit from Internap or some other route aggregator and Level3 may have lost allot of money over the years so it should come as no surprise that they are at it again. This time with some method to their madness, they baiting customers and will up sell them on value added services or give the baseline product away for free but demand a share of the customer's revenue generated as a result of distributing content. In essence subsidizing the delivery costs of that content.

Questions that come to mind about Level3's shuffling of the product pricing boxes: How much is their storage going to cost? What about those companies who need some DRM functionality for their downloads? How much with that cost? What about custom players or layered advertising? How much with they charge for that and what percentage of the advertising revenue will they require you give them for using their CDN?

I read their new delivery pricing works out to about $11 per Mbps equivalent. Not a bad rate, but certainly not as low as some of the pricing from other players in the space. And definitely not free, yet. It will be free sooner than later because it will be subsidized by advertising dollars.

First it may be helpful to explain the difference between transit of data and distribution of data. Everyone on the internet has transit. It is the ability to send packets from one place to another. Inherently there is some form of intelligence built into transit because it is built into the core backbone routers of the major carriers and isps.

Example: you're sitting at your computer on Sunday morning checking your fantasy football scoring on the Sportsline website. You live in CA and have Comcast as your ISP. Sportsline is hosted in FL at the Terramark datacenter(not sure if this is actually the case and am using as an example only) and connected to the Internet in Terramark via Sprint. When you type in the url and hit return a request gets sent from you computer out your cable modem and onto Comcast's network. Comcast's routers see the IP address of where that packet is supposed to go, looks up in it's routing table what would be the quickest way to get that data off of it's network and onto the Sprint network. Since quickest doesn't always translate into best and in this case only translates into Comcast not wanting to carry the cost of that packet on it's network your experience is subject to all sorts of hiccups along the way. That first packets eventually gets to FL and Sportsline and when it does, the same process, only in the inverse order, happens again. The updated score makes it's way back to you in CA and all is well. In this instance the experience wasn't all that bad but the information you requested was quite small in volume of data required to get you that score. Imagine if, instead of the updated score, you were requesting the infamous Paris Hilton and Rick Soloman video. The file size of that video is exponentially larger than that of the score of the fantasy game and the cost to Comcast and Sprint is also exponentially higher to deliver than the score. Fortunately for you, Rick Soloman was out to make some dough and wanted the user experience to be similar to his own experience so he hired a CDN to help ensure the quality of the experience remained tip top :) Important detail to note, CDN's all buy transit and most buy it from multiple upstream providers. In this made up example, Rick was uhttp://www2.blogger.com/img/gl.link.gifsing ACME CDN who was a new CDN. Unlike Akamai, ACME was built from the ground up to deliver large video files and wasn't too concerned with the last mile. As such, they had a smaller # of nodes placed in carrier neutral data centers on the two coasts and a couple in the South and Midwest. Since ACME isn't a web hosting company, Rick needed to contract with a second vendor to host the rest of his homepage. Rick is a sharp cat and a penny pincher so he used Amazon's S3 file storage service which would itself have been sufficient had the video been of Rick and anyone but Paris. He understood S3's limits and since he was out to make add to the stack of paper he was blowing on Paris, he only relied on Amazon for the non revenue generating files. When you click on the 'watch' button on his site the url where it sends you is not on Amazon's infrastructure but on that of ACME's. Similar to the fantasy scoring example, Comcast's routers determine that the destination address is part of address space supplied to ACME and easily accessible via SBC who happens to be one of Comcasts upstream providers. Comcast dumps this packet onto SBC. Unlike Akamai, ACME's technology doesn't rely upon multiple DNS queries to determine where the originating or destination IP address is physically located. ACME uses anycast protocol/technology which allows it to have the same ASN at all of it's nodes. Another difference with ACME is that they aren't using a cacheing setup but instead are acting as the origin for their customers files. This allows them to guarantee that 100% of their customers files to be distributed will always be on 100% of their nodes. When using a cache, this can't be the case as only the most requested files remain on the node for a matter of time and then they're purged. Its great if you're popular but not so great if you're not popular of are very private data but depend on high performance. When SBC gets that packet it see's the AS it's destined for and dumps it as soon as possible. In this case that hap pend to take place in the same building, Equinix in San Jose. Both ACME and SBC are colocated at Equinix so there was minimal cost to SBC to deliver that packet which makes them happy to deal with ACME. Can't say the same for their dealings with Akamai because Akamai's nodes are out at the edges of SBC's network, in the CO's which would mean SBC has to keep those packets on it's network far longer than their ideal. Amazon's S3 cloud is located in VA and had Rick not used ACME for the origin location of the video, a similar process for delivering the fantasy football score would have occurred with the deliver of the video. Fortunately for you, Rick's decision to use ACME results in your viewing the video almost instantly and not waiting around for the packets to go back and forth from the West coast to the East coast.

So what does this have to do with Level3 pricing or the pricing of the CDN market in general? It highlights how much of a difference adding an origin storage component to a CDN can make in terms of route miles. If Level3 can charge customers more money for using less route miles then all the better, it may even make sense to give transit away if they can make up that $11/Mbps by making customers believe they're being so cutting edge with pricing and product positioning when in reality they're stroking themselves by driving more efficiency on their network.

Ohhhhh, if that were only the case. I guess theory sounds good on paper but reality is what it is.

The Data Center Cheat Sheet - What exactly are we dealing with?

It may be useful to go through a brief overview of Internet datacenter market history to properly appreciate todays market dynamics so bear with me if this is old news or a regurgitation of a not so happy time. Those times build character though :)


Over the past few years the Datacenter market has experienced a shift in power as it relates to the Datacenter Vendor and customer or prospective customer realtionships. This is a function of an imbalance in supply and demand. From 2000 to early 2005, it was a buyers market for colo and buyers played vendors off of each other to get the very best deals they could. And they were quite successful in getting the often desperate vendors to strike deals that were well below being financially healthy or sound. From the vendors perspective, they were just happy to get customers in their datcenters. After all, they had rent to pay to their landlords and sitting inventory that is not generating any money is worse than selling that inventory for anything greater than zero. Allot of poor pricing decisions were made during this window of time but they(pricing) weren't the only questionable attributes of the deals that went down during this period.

The bigger thorn in the side of most of these deals was related to what these customers were allowed to install in each of the racks or cages in the colo's. Remember the time and put yourself in a vendors shoes for a minute. You're negotiating with eBay or some other large retailer and just the thought of signing this customer makes you forget the notion of profitability. At this point stopping some of the bleeding will be a step in the right direction and as such you agree to give ebay the best rack pricing you've ever given anyone and don't put any parameters around how much power they can install and consume. Secretly you're really hoping they over provision power because that is money in your pocket that helps to offset the low rate on space you've agreed to. Sidebar definition: Over provisioning power is the scenario whereby a customer provisions 60 amps of primary 208v power(as an example) and only consumes 20amps of it. The customer pays the vendor for the full 60 amps but the vendor is only on the hook to the utility for what it uses, in this case 20amps. That is 40amps of profit right? Yes, at that particular month it was. This was quite a common situation and more often than not it was because the customers were ordering their colo configurations based on what their equipment required at full load. The issue here was that nobody was using the equipment to anywhere near capacity.

Slowly but surely the economy crawled back up and to the right(on a graphical basis) and with it came increased usage of the internet, ubiquity in broadband access, storage prices plummeting and innovation in usage of the internet in general. With the economy coming back more people were employed and they sure surfed the net at work(I think it would be intersting to see a study done on productivity output of employees with internet access and employees without it), more people had disposable income so they could afford the DSL or cable modem which allowed them to get further faster in their online worlds and gave them new ways to interact with one another via social networks which blended and intermixed with their real world lives. All of a sudden that steady 20amps of power consumption start to creep up. And up. And now frighteningly up. Up to the point that, as one VP of Ops of a big player in the space and who shall remain nameless, said,"this place could blow at any moment"

IMO, this was the point which the tables turned in favor of the vendors. By now the supply and demand was getting back to a state of equilibrium and it forced the datacenter vendors to do what I refer to as 'robbing Peter to pay Paul.' In order to fully grasp that notion you must understand what a datacenter really does. At the end of the day, a datacenter provides space, power and environmentals to it's customer sets. That is it. Datacenters don't provide managed services, service organizations do. Datacenters don't provide CDN or transit, ISPs and CDN's do. Datacenters don't provide storage, storage providers did that. We're talking about what the physical datacenter provides. Space, power, environmentals and physical security. Some may argue that these vendors provided interconnectivity and the vendors did but that was an added service layer that in actuality doesn't need to be a product of the vendor but could be the product of anyone or nobody(if it was free). When a datacenter is built you start out with a shell of a building and an amount of power that you can get delivered to that building. With that shell floor plan and that maximum amount of power available to you, you develop an overall layout of where things will go. Things being chiller, cooling towers, air handlers, generators, batteries, diesel storage, water storage, shipping and receiving, ingress/egress points, different authority levels of access, security and so on. You don't make these decisions without first knowing how much power you can get because there is a direct correlation to that amount of power and how many pieces of the Mechanical Electrical infrastructure plant will be required and how much square footage they'll occupy in the building. Long way of saying there is a finite amount of power and environental resources available for consumption. The standard increment or unit of measure in the market is either a rack or cabinet(42RU of actual space) or a sq ft. Each rack takes approx 20 sq feet of space on the datacenter floor. In order to forecast revenue, the datacenter operator simply takes the total sq footag of raised floor and divides by 20 sq ft to gt the # of available rack spaces they can sell, giving them some ability to forecast revenue. And they did forceast revenue based on these simplistic equality assuming assumptions. So if you have 50k sq ft of space you can sell 2500 cabinets. At $800/month per cabinet you'll generate $24MM in annual revenue. Sounds like a good plan right? The issue isn't it's simplicity but rather that it is only one piece of it, space. What about power? If you have 5Megawatts of power available for customer consumption across that 50k sq feet, you have a datacenter built to 100 watts a foot. If you have 7.5Megawatts of power available for consumption in that 50k sq ft, you have a datacenter built to 150 watts a foot. 10megawatts and you have 200 watts a foot. And so on.


Taking a step back, remember the example of the customer who was allowed to install the 60amps of 208v power in that single rack or those 20 sq ft? 60amps of 208v power in 20 sq ft equals about 500 watts a foot. Remember the notion of a finite amount of power coming in to the building and the linear relationship between power and the amount of space required for mechanical gear? That is because when delivering the power to the customers, the customers consume it with via the hardware infrastucture and in doing so, that hardware gets hot and gets hot quickly. Hence the beefy AC's that are required in datacenters. The same concept of the division of resources is carried over and applied to environmentals. We still don't have a global standard unit of measure for the industry because each building has different attributes and a customer may achieve higher utility in one vendors rack vs a different vendors rack because of the difference in the amount of available power in that rack. For this reason comparing Equinix rack pricing to Terremark rack pricing is useless unless you know the power per sq foot in each of their buildings. What point is there is trying to get Equinix who for examples sake has built out a datacenter at 200 watts a foot and is offering racks for $1000 each to lower it's rate to the $700 monthly fee that Terremark is offering in their datacenter which is built to 100 watts a foot. Don't you see what a screaming deal you already have with Equinix? To get that same functionality or utility at Terremark would cost you $1400 a month per rack.(Vendors and associated #s there are meant for expample purposes only). Circling back to the example earlier of eBay over provisioning those 60 amps of power or 500 watts a foot in the 100 watt per foot designed facility and you quickly realize that you, as the vendor gave up 5 racks of space and associated revenue for everyone one rack of space that eBay pays for. And pays for at the lowest rate you ever did. The deal is 5X worse than you thought. Not only that, but the perception of your company to a stranger walking in to your facility is that you are struggling because your datacenter is only 20% occupied spacewise because those first 500 racks that ebay installed consumed all of the power and cooling resources. Now imagine your the vendor who didn't catch this overprovisioning issue until you had oversubscribed your mechanical plant by a factor of 2 or 3X and you have all customers usage creeping up simultaneously. What do you do then? You say, "this place could blow at any moment" :) Those of us who lived through those types of situations and conditions will never get in them again. The first time around can be chalked up to ignorance. The second time would only be stupidity. This thought is evidenced by the hard lines the vendors take today as it relates to placing limits on the amount of power per rack they will allow their customers to install.

Taking the example from earlier with ebay using the entire pool of resources in 20% of the space of in the building and you can view it one of two ways. The first being that the supply of available space just shrank by 80% or the demand for space just increased by a factor of five. The market adjusted itself and the tables turned in favor the datacenter vendors and shows no signs that it will revert back to it's old ways. Sure, you hear allot about new datacenters being built today but remember, there hasn't been any signficant investment in this space in about ten years. During those ten years, computing clusters have gotten physically smaller and financially cheaper while increasing in performance. All of this resulting in more power consumption per rack unit, doing more in less space but with no change in the relative volume of an amp of power. Meaning the computers got more efficient in both performance and amount of space the physically take up but the power is what it is. And that is a study of physics. Efficiencies aren't a part of power, they're a part of those things that use power. Wrapping this up, the market has experienced all sorts of technological progress on hardware and software piece of the equation allowing users to pack more in to less but that less consumes exponentially more power than that more did in the previous scenario. The most scarce resource of a datacenter is power. And that means cooling too.

datacenters, data center, watts/ft, kw, kv, power, density, hvac, colo, equinix, savvis, terremark, internap, global crossing, exodus, amazon, salesforce.com, ebay, efficiency, amps, volts,