Google Net - All things considered, a cheap means to an end

So they're taking it one step further than I thought and are taking the fiber all the way to the home.

-Makes sense as they'll have zero dependence on any CLEC, RBOC, MSO or local transport layer. They'll own the traffic from the starting point to the end.

-This gives incredible scale and opens tons of opportunity to exploit mindshare to drive more revenue.

-Google building this network is kinda like the equivalent of the inverse of the Comcast purchase of NBC. Instead of NBC paying Comcast fees to carry it's traffic, NBC now has a free ride to the TV sets of Comcast subscribers. Similarly with GNet, Google now has a free ride to GNet users/subscribers/testers and bypasses paying transit and transport fees to telcos, etc.

The big question is how does this fit into their network neutrality stance. Will google let other transit providers (aka ISPs) ride this last mile to the house?

Why should they when they are the ones spending the money to lay the fiber?

My gut tells me that the answer is yes but there will be a cost. That cost could be in the form of monetary compensation from the ISP to google or that cost could be that Google gets visibility into the usage of the ISPs customers. That visibility and the historical data resulting from it is likely far more valuable to Google than the revenue they could generate by selling the last mile to alternative ISPs.

Envisioning this one step further, by connecting their network to all private networks run by telcos, isps, msos, ilecs etc, Google can become the arbitrator/clearing house of peering (traffic exchange) between consumers and traditional network operators. Creating a true, on demand, utility based connectivity model. Like a cloud for network service. Like bringing BGP to the masses by enabling it at the core instead of the edge.

Can they have their cake and eat it too? Time will tell.

Web 3.0 Just Kicked Through the Door

In the playbook for Google's world dominance there is one play still in the developmental stage:

GoogleNet - the one missing link. No pun intended with the missing link part. Link....network... get it?

The posting below is a repost from February of 2007. I'm reposting because I think it is just as applicable today as it was then.

Couple quick thoughts before my paste:



- the amount of data on every persons habits that google will be able to manipulate and exploit is scary. the Chrome OS and Android just gave google 5 times the access they had to me before. Why? Android was built to run on devices and I can't stand windows crashing every hour so i'll gladly switch to a more stable android. not to mention its free. Android will be the OS for my mobile phones, set top boxes, home entertainment systems, appliances (refrigerators, ovens, vehicle management and entertainment systems, home automation devices (remember google's forays into the home automation and electrical smart meter markets?) and probably, in the not too distant future, they'll manage our terlits too. Thats a shit load of visibility they'll have in to the patterns and habits of connected people. Again, no pun intended :)

- will the Feds just sit back and watch as Google take it deeper than Ma Bell ever dreamed possible?






They are giving away everything else, why not connectivity too? Its all about efficiency right? Operational efficiency, risk mitigation efficiency and customer efficiency. Efficiency is the driving force behind Googlenet(gnet). What is GNet? Google's foray into the ISP business. This 'business' for Google is the means to an end. At the end of the day Google sells advertising and they'll use whatever means necessary to do that, be it building an OS and giving it away for free or by providing free connectivity so that free OS stays online 24x7. The more interaction we have with devices the more impressions google has to sell to advertisers.

The gnet hypothesis: Bandwidth costs have fallen to a level that the advertising revenue more than subsidizes the cost of the network. We are in the very early stages of a true 'gloabl village' as Marshall McCluhan called it. The cost structure for a traditional ISP like PacBell DSL..errr AT&T, comcast, etc to supply services to the residence is around $40 per month and trends down as they grow because they get cost scale.... in the network world, the more you buy the less it costs.

goog will be placing a bet, and a very calculated one, that advertising $$ will not only subsidize operational costs of providing free services(isp, voip, office, etc), but will exceed them.

Owning the customer's network routes from end to end(being the ISP) provides Google with private infrastructure platform for delivering customized content and adverting to each and every one of the people using their service. this delivery platform is always on and knows where you go, what you type, where you live, who your friends are, what files you have downloaded, what you look like, and whatever else they add on to their services. So when Johnson & Johnson or GE or Proctor & Gamble or Coca Cola or Pepsi is planning their media buys for the next year do you think they'll purchase advertisements on radio, television, print or the 4th network(Google)? based on the ability to target a specific population that has certain attributes you desire, the choice is clear...you pick google. Why? because you know that your marketing msg is going to a qualified prospect as opposed to the traditional 'shotgun' approach. plus, you can get results in realtime and tweak your msg if its not working in real time. with the other three media you are somewhat ratholed into trusting some third party for ratings that may or may not even reach the people that you want it to. by the time you figure this out a slew of things can happen...some good some bad but why chance it when you don't need to.

At the end of the day, Google is building a traditional media killer and the funny thing is..actually not really funny but kinda, that the writing is on the wall but nobody seems to believe it. I do. @Home Network had this vision but couldn't pull it off because the cable co's couldn't get their heads out of their rear to see the opportunity that was sitting right in front of them.

goog already has deals in place and likely in the works with the producers of original content which will allow their users access to that content. if i'm a content producer where would i want my content to be seen? ex. let's say you are the producer of The Office and goog offers you the ability to place your content on their network so that it can be viewed by anyone, anywhere, anytime and offers a revenue share or some other creative structure around it such that you know your worst case scenario beforehand. simple choice right? sure abc or cbs might offer an upfront fee in the form of $$ per show but the audience is limited, the timeslot is finite and in order for someone to view it, they have to purchase cable tv or satellite or whatever whereas on the google network you content would be globally accessible and the broadband access which replaces the cable or satellite tv service, is free to the masses. additionally you can develop complimentary services that engauge your viewership such that you are able to really develop a community around you content as opposed to content around a community.

What does this have to do with being an ISP? Everything, why do you think there is such a brouhaha over net neutrality? Without connectivity none of this is possible and with the right connectivity, all of it is possible and defendable from the threat of new competition emerging to pose a challenge.

The internet hasn't changed anything when it comes to the bare bones media model. The driving force of media is and always will be advertising. Without it, there would be no televsion, radio, print or internet. It doesn't matter if its old world or new world, it's still dependent upon advertising and gnet is to google what airwaves were to ABC, NBC and CBS, something to exploit in order to sell advertising.

Data Centers are the Economy

The dependence on the data center today is far deeper and wider than it was in 1999. How could it not be? The Internet is no longer just another source for information threatening print, radio and television. It is THE source for information, THE source for education, THE source for communicating with individuals or to populations, THE source for commerce and trade, THE source for entertainment and THE source for a recession proof economy. At the heart of it all is the data center. The data center provides the platform which is enabling a more equitable distribution of wealth across a global stage.

Assuming this is all true, which I certainly believe it to be, there should be an epic flow of resources directed towards the buildout of the data center platform. Guess what...there isn't. The largest investments being made today are coming from Microsoft and Google. Give them credit. For all the grief the two of them take, they really "get it." And they've been rewarded financially for "getting it." But what about the innovation that is a result of the fresh ideas and new ways of doing things? Google and Microsoft no longer possess such innovative ways yet they own a large percentage of the platforms which enable such innovation. How many startups challenging MSFT, GOOG and AMZN are going to be comfortable hosting their secrets on the very companies they are trying to dethrone? I doubt too many.

There is a huge misconception that a glut of data center space is on the horizon. The fact is that perception is ill conceived and those who believe it and make investment decisions are ill informed and quite possibly passing up a once in a lifetime investment opportunity. Don't misunderstand my point of view, yes, there has been a fair amount of new data center builds announced and in some cases started but if you go back and look at some of the recent ones they are rarely don't on spec...ie, the risk is minimized and the inventory rarely hits the retail supply. Take a dive into the SFBA market and examine Digital Realty Trust's two most recent builds in Santa Clara. Both were pseudo started on spec but were both completely sold out before being finished. In both cases they were leased to single occupants...ie, they got two new customer orders out of them. Not two new customers as both orders were singed by existing customers, but two new orders. Hardly a speculative build when you know your existing customers are about to hit a wall in terms of their available capacity and you can lead them down a golden path to expand in facilities operated by a vendor they already do business with....oh and which happen to be the only place on the West Coast where they can walk into such a situation. In the same market and right around the corner from DRTs two sold out locations there is Equinix, formerly the leader in speculative builds. Equinix is expanding its existing Santa Clara facility by around 40k sq ft and if it chose to could have the entire thing sold out today. Since their target customer isn't a 40k ft requirement and since they're in an enviable financial position, they can be choosy about who they sell the space to and find customers who will pay them top dollar for their product. This definitely won't be a startup as they are too cost sensitive and aren't worthy of extending large amounts of credit to in the form of inventory....Equinix has been there and done that and learned from the past. What about the rest of us? What about the major consolidation at both the state and federal government level which is far greater than anyone anticipates? What about the major efforts going on in enterprises across the US which all require data center facilities which are far more robust than what is currently available to them?

If data center capacity is not available for these and many more types of requirements it is a serious threat to the growth of the economy and only positions the big guys more favorably than anyone should be comfortable with.

Advanced Data Centers, Inc. 50 Megawatt Sacramento Project

For those of you that have a blog or maintain a website you probably are familiar with web analytics. I was first introduced to web analytics in my days at Equinix where I managed the Omniture relationship. When I first began working with Omniture they had a couple racks of servers. By the time I left Equinix, Omniture was close to 1000 racks of equipment spread across the globe. This happened over the course of a couple years. It didn't take a dummy to figure out they were onto something. A couple months after I left Equinix I started blogging and was introduced to web analytics from the point of a customer/user. I believe analytics is one of the most powerful elements of online behavior. Since the internet is subsidized by advertising dollars analytics plays a huge role. But this post isn't about analytics, it's about, as the title suggests a new 50MW data center project underway in Sacramento, CA. 50 Megawatts is the most power at any data center site that I have ever come across. This project has been and will continue to be the recipient of a good portion of my attention for quite some time....or until we are out of space and power.

After spending 7 years at Equinix I thought I was done with data centers. So much so that it made my head spin even thinking about them. Shortly after leaving, it dawned on me that the data center market is unique in that there are natural barriers to entry in the form of extremely high cost of admission. Some people claim that data centers are being commoditized and with all the new construction taking place this line of thought is getting more attention. For those of us building data centers this is exactly what we want....that is of course as long as it's not the attitude of an investor or banker.

It's positive for us in that by virtue of it's negative connotation, it naturally scares new entrants away, creating additional strains on supply. It's also good because it makes it very difficult, if not impossible, for a new entrant to raise the necessary capital to build out a large scale data center due to construction costs of approximately $1500 per square foot. That cost is enough to scare most people away alone. There are quite a few instances of major corporations in the US forgoing the outsourcing model of colocation and becoming site operators themselves. Without huge augmentation, very few companies have the scale and resources to make this work over an extended period of time. Not impossible by any stretch but certainly challenging. Probably the most challenging element in getting total executive buy-in/support takes place when the CIO meets with the CFO, telling him that he needs $250MM to build a new data center that will support their IT requirements for the next ten years. More often than not, the CFO recalls a similar discussion 18 months ago in which this same CIO was asking for budget to outsource the same type of requirements, albeit less capacity, to a colo supplier. Once the CFO realizes this isn't a prank and that this guy is serious he collects himself and pours a stiff drink(it's a late night at the office in this example so its all good.)

When we embarked on our new venture, we had these types of discussions in mind and saw an opportunity to provide a service which would give the CIO the resources to support their IT customers and, at the same time, give the CFO a heart attack at every budget meeting.

I'm getting sidetracked so I'm going to get back to the analytics component. I find it very interesting to take a close look at how people get to this blog. Some get here by entering terms on the various search engines while some get here by clicking on links on other blogs or news sites that reference this one. And for some, I have no idea how they wound up here. It has been enlightening to see what people search for when they go to google, yahoo, or whatever search engine they use. I've found that a good portion of the folks who wind up here search for common terms that we throw around every day in the data center world...things like: watts per sq foot, kva to kw conversion, 60amp 208v circuit, Equinix rack price, Savvis data centers, 365main outage, etc. You get the point. The analytics service I use allows for me to see the address and domain of these readers network and has given me some insight to what IT people within these organizations have on their plate...or what keeps them up at night. Then again, I could be so wrong and off base I may as well be fire on an frozen pond. Either way, people who wind up here on my blog appear to be in the process of determining how to address their data center requirements.

Without overstepping the boundary of turning this blog into a sales pitch, I really believed the company behind the aforementioned data center project, Advanced Data Centers(ADC), may be a valuable partner to those folks in need of capacity for their IT gear. As such, what better way to reach them than to discuss it here.

If you're one of those folks or think you will be soon, I invite you to visit the ADC website to see what we're up to and if it looks intriguing drop me a note either on this blog or via the form on the contact page of the website.

Level3’s smoke and mirrors

Level3 didn't slash pricing for CDN service with their recent 'same price as transit' marketing ploy. If anyone slashed pricing, Amazon did with the introduction of their S3 storage. All Level3 did was imply their baseline CDN service is no better than their transit....heck, at least they admit it. A couple points to note. Baseline CDN service really is transit or no better than transit from Internap or some other route aggregator and Level3 may have lost allot of money over the years so it should come as no surprise that they are at it again. This time with some method to their madness, they baiting customers and will up sell them on value added services or give the baseline product away for free but demand a share of the customer's revenue generated as a result of distributing content. In essence subsidizing the delivery costs of that content.

Questions that come to mind about Level3's shuffling of the product pricing boxes: How much is their storage going to cost? What about those companies who need some DRM functionality for their downloads? How much with that cost? What about custom players or layered advertising? How much with they charge for that and what percentage of the advertising revenue will they require you give them for using their CDN?

I read their new delivery pricing works out to about $11 per Mbps equivalent. Not a bad rate, but certainly not as low as some of the pricing from other players in the space. And definitely not free, yet. It will be free sooner than later because it will be subsidized by advertising dollars.

First it may be helpful to explain the difference between transit of data and distribution of data. Everyone on the internet has transit. It is the ability to send packets from one place to another. Inherently there is some form of intelligence built into transit because it is built into the core backbone routers of the major carriers and isps.

Example: you're sitting at your computer on Sunday morning checking your fantasy football scoring on the Sportsline website. You live in CA and have Comcast as your ISP. Sportsline is hosted in FL at the Terramark datacenter(not sure if this is actually the case and am using as an example only) and connected to the Internet in Terramark via Sprint. When you type in the url and hit return a request gets sent from you computer out your cable modem and onto Comcast's network. Comcast's routers see the IP address of where that packet is supposed to go, looks up in it's routing table what would be the quickest way to get that data off of it's network and onto the Sprint network. Since quickest doesn't always translate into best and in this case only translates into Comcast not wanting to carry the cost of that packet on it's network your experience is subject to all sorts of hiccups along the way. That first packets eventually gets to FL and Sportsline and when it does, the same process, only in the inverse order, happens again. The updated score makes it's way back to you in CA and all is well. In this instance the experience wasn't all that bad but the information you requested was quite small in volume of data required to get you that score. Imagine if, instead of the updated score, you were requesting the infamous Paris Hilton and Rick Soloman video. The file size of that video is exponentially larger than that of the score of the fantasy game and the cost to Comcast and Sprint is also exponentially higher to deliver than the score. Fortunately for you, Rick Soloman was out to make some dough and wanted the user experience to be similar to his own experience so he hired a CDN to help ensure the quality of the experience remained tip top :) Important detail to note, CDN's all buy transit and most buy it from multiple upstream providers. In this made up example, Rick was uhttp://www2.blogger.com/img/gl.link.gifsing ACME CDN who was a new CDN. Unlike Akamai, ACME was built from the ground up to deliver large video files and wasn't too concerned with the last mile. As such, they had a smaller # of nodes placed in carrier neutral data centers on the two coasts and a couple in the South and Midwest. Since ACME isn't a web hosting company, Rick needed to contract with a second vendor to host the rest of his homepage. Rick is a sharp cat and a penny pincher so he used Amazon's S3 file storage service which would itself have been sufficient had the video been of Rick and anyone but Paris. He understood S3's limits and since he was out to make add to the stack of paper he was blowing on Paris, he only relied on Amazon for the non revenue generating files. When you click on the 'watch' button on his site the url where it sends you is not on Amazon's infrastructure but on that of ACME's. Similar to the fantasy scoring example, Comcast's routers determine that the destination address is part of address space supplied to ACME and easily accessible via SBC who happens to be one of Comcasts upstream providers. Comcast dumps this packet onto SBC. Unlike Akamai, ACME's technology doesn't rely upon multiple DNS queries to determine where the originating or destination IP address is physically located. ACME uses anycast protocol/technology which allows it to have the same ASN at all of it's nodes. Another difference with ACME is that they aren't using a cacheing setup but instead are acting as the origin for their customers files. This allows them to guarantee that 100% of their customers files to be distributed will always be on 100% of their nodes. When using a cache, this can't be the case as only the most requested files remain on the node for a matter of time and then they're purged. Its great if you're popular but not so great if you're not popular of are very private data but depend on high performance. When SBC gets that packet it see's the AS it's destined for and dumps it as soon as possible. In this case that hap pend to take place in the same building, Equinix in San Jose. Both ACME and SBC are colocated at Equinix so there was minimal cost to SBC to deliver that packet which makes them happy to deal with ACME. Can't say the same for their dealings with Akamai because Akamai's nodes are out at the edges of SBC's network, in the CO's which would mean SBC has to keep those packets on it's network far longer than their ideal. Amazon's S3 cloud is located in VA and had Rick not used ACME for the origin location of the video, a similar process for delivering the fantasy football score would have occurred with the deliver of the video. Fortunately for you, Rick's decision to use ACME results in your viewing the video almost instantly and not waiting around for the packets to go back and forth from the West coast to the East coast.

So what does this have to do with Level3 pricing or the pricing of the CDN market in general? It highlights how much of a difference adding an origin storage component to a CDN can make in terms of route miles. If Level3 can charge customers more money for using less route miles then all the better, it may even make sense to give transit away if they can make up that $11/Mbps by making customers believe they're being so cutting edge with pricing and product positioning when in reality they're stroking themselves by driving more efficiency on their network.

Ohhhhh, if that were only the case. I guess theory sounds good on paper but reality is what it is.

Sun's Blackbox

Get your mind out of the gutter, I'm referring to their portable datacenter. I was able to attend one of Sun's introductory briefings today in Menlo Park. When Jonathan Schwartz first announced this as a product I was very skeptical and threatened. Skeptical because these containers are 160 sq ft and can support a 200kw draw. That is 1250 watts per foot, albeit very isolated. And threatened because of the potential disruptive effect these new devices could have on the traditional datacenter market, my livelihood. Kinda.

I'm still skeptical but not as much as I was. I'm definitely not threatened, not because I don't believe in the viability but because the two are more complimentary than exclusive.

There are a few kinks to be worked out or how shall I say, items that are quickly set aside during their presentations but what did you expect? Marketing, marketing. Anyone know Al Hops?

Anyway back to this Blackbox.... A couple issues to note:

- these are NOT stand alone units. they require:


- multiple high voltage power connections in a minimum n+1 config - approx 250kw of provisioned primary power(these aren't connections you just run an extension cord f or. these are serious high voltage connections and as such require a serious infrastructure plant to get the connections down to the voltage required by the box. you don't call PG&E up and order one of these. typically this will be a branch on a larger power grid and in the datacenter world can be likened to a 12kv branch to a PDU.

- Cold water - Blackbox units require a cold water feed to support cooling off the payload, if you will. . To support 200kw of draw is approx 30 tons of chiller for these Blackboxes. The chiller doesn't come with the Blackbox and doesn't fit on or in one. Infact, a 60 ton chiller, enough capacity for 3 boxes, is about the size of a box itself. Chillers require power to produce cold water and you don't just plug a chiller into your wall outlet and be on your way. It requires the same or similar type of connections as the Blackbox, hi voltage, hi capacity circuits.

- Water Supply - HVAC systems will lose water to condensation, evaporation, leaks, overflows, etc and that water needs to be made back up to ensure smooth sailing. Maintaining N+1 design, you need two supplies of water from seperate suppliers. One is obviously your regular water supply but what about the second? dig a well like most datacenters do?

- UPS systems. There aren't any. Seriously. So that should tell me who the target customer is. Someone who doesn't care about uptime? The why the hell buy all this crap? why not host it on Amazon S3 or MediaTemple? Who doesn't care about uptime? Google is the only company I can think of, actually amazon too, who wouldn't care if they lost 8 racks of servers. I just don't think Sun is far enough along to have a solution for UPS that doesn't make you take a step back and say, 'wait a second, where the hell am i going to park five* tractor trailers so i can operate my 24 racks?' * 3 actual Blackbox container, 1 container for Generator and batteries and one container for the chiller.

I sound like I'm bagging on Sun but I'm not really. I like the idea and know it's a definite winner in niche applications such as military use, natural disaster use, isolated locations where it can be airlifted in and so on.

The thing is, if Sun owned the entire market for those specific applications it still isn't going to get Sun where it needs to be, it's just too limited in size. Sun needs to find a way to make these Boxes the defacto standard choice when a company begins evaluating datacenter options. That or sell the concept to the colo vendors by delivering them value by showing that the Boxes can compete economically with a standard raised floor environment. Coincidentally, just like a regular datacenter, in order to support a few of these boxes you will need a significant MEP resource which is essentially the bread and butter of a datacenter and datacenter operators are experts are managing MEP. Its a nice fit.

I liken the potential of Blackbox type architecture to what consumers are using Amazon S3 grid or google's own infrastructure(googleOS) for, a shared IT resource that supports unique data for each user and leverages commonalities among users. everything is virtually connected and resources are shared so if one goes down it doesnt matter yet the performance benefits of close proximity is omnipresent.

Cost. The fully built out container(without the computers, chiller, generator and truck or helicopter to transport it) currently costs $500k to build. Sun eluded to the price point of $250k as one which they're shooting for. $250k for 200kw isn't a bad deal. Equinix spends about $25k per rack or $1000/sq ft for a 2.5kw rack. In gross #'s Suns Box looks good at $1200/kw on the Box while a traditional datacenter, per Equinix's rough costs, comes in at $10,000 per kw. I don't know what the cost of the chiller plant and elctrical switches, etc would be but imagine it can't be more than 60% of the total costs of construction of the traditional so add another $6000 per kw and mutiply that sum, $7200, by the number of kw draw and you get your total cost for the Box and the supporting MEP gear. In this case it is $1.4MM for 200kw of datacenter equivalent. For Equinix, it would cost $2MM+

Lots of potential with this product but in order to be mass adopted it needs to demonstrate an economic benefit in addition to the obvious operational ones.