From Big Transit to Big Platform

From the early days of the Internet, there’s been a clearly identifiable structure that rather contradicts the public image of it as a seamless mesh of interconnection. Edge networks were served by ISPs, who in turn relied on the major, frequently American, transit carriers for their upstream connectivity. This structure emerged from the development of the NSFNet in the 1980s and 1990s, in which the US and European NRENs (National Research and Education Networks - JANET in the UK, GEANT in France, NSFNet and then Internet 2 in the US) first interconnected university and private R&D organisations, and then interconnected to each other using a small group of telcos’ wholesale service. Later, the arrival of commercial Internet backbones from 1994 and the creation of network access points (NAPs) cemented this structure in place.

A small elite of operators relied entirely on peering with each other, the so-called Tier 1 carriers; if you weren’t Tier 1, you were a customer. Tier 1 domination was based on two scarcities - that of backbone connectivity, and that of interconnection. The first was undermined by the massive investment in dark fibre of the .com boom, and then overwhelmed by the second wave of submarine cable investment in the late 2000s. The second was undermined by the growth of Internet exchanges - in contrast to a NAP, where lower-tier carriers interconnect with Tier 1 carriers, an IX is a facility where networks of any size exchange traffic, usually a membership organisation. Although the first IXen appeared almost as soon as the restriction on commercial interconnection with the NSFNet was lifted in 1994, this structure survived into the mid-2000s.

Now, though, it’s gone. In 2007, all the top 10 networks by ATLAS traffic measurement were global transit carriers - all except for Cogent and Telia were formally Tier 1 in the sense of being transit-free. The list looks very much as it would have done in 2002; AT&T, Abovenet, Sprint, Global Crossing, all there, Verizon would have been UUNet and MCI. Today, although Level(3) and GBLX are still on top, Google is number three and Comcast Cable number five. The impact of change is visible in terms of pricing; transit is becoming a super-dumb pipe product.

Google, meanwhile, now accounts for almost as much Internet traffic as Level(3) did two years ago! The move towards direct peering interconnection between content and eyeballs is well underway. And it’s also notable that YouTube is disappearing as a separate entity for internetworking purposes - subsumed into the Google infrastructure.

CDNs - Supertankers of the Internet

But it would be profoundly wrong to conclude that Big Content has won out over Big Telecom, and that the plausible talkers of the 90s were right about content being king. You will look in vain for content owners in the top 10. Rather, the key actors in the new look Internet are the Big Platforms. Google is one; the others are the major CDN operators. The top five content delivery networks - CDNs - now account for 10% of global traffic. In fact, because ATLAS only tracks interdomain traffic, it’s probably closer to 15% - they estimate that three-quarters of Akamai’s traffic is intradomain, between its edge servers and hosts on the same local network. This is, of course, the point of Akamai’s existence.

This has a crucial role in another trend ATLAS identified; in 2007, the top 30,000 ASs (Autonomous Systems - roughly, individual networks) accounted for 50% of global traffic. Today, the majority of global traffic is heading to or from the top 150 networks. But this doesn’t imply a hierarchical structure like that of the old days; rather than going via 111th St NYC, UUNet, and LINX to reach content, users get it from local CDN servers. The average hop count, a measure of directness and routing complexity, has fallen to 3.5.

Content is king; but distribution is King Kong, and there’s more to distribution than just pipes.

Everything is a Web site: our client/server Internet

Another major trend is the concentration of traffic into the World Wide Web. Traffic on HTTP port 80 is by far the biggest category and the fastest growing. Rather than fancy P2P systems, the leading distribution systems for Internet video are HTTP and Flash streaming - i.e, just like YouTube.

Although P2P traffic seems to be falling, however, it’s not going away; it’s also hardening its defences by using strong encryption and selecting its ports at random, thus defeating characterisation either by port category or by deep packet inspection.

Comcast: Wholesaler

If P2P is not what it used to be, surely this means that the residential eyeball networks are more downstream-heavy than ever? You might imagine we’re heading for a content is king Internet, where the big media industries (or rather their logistics partners, the CDNs) shovel movies at a passive user population via a downlink-heavy ISP community. But, fascinatingly, this isn’t happening; one of the most TV-minded of ISPs, US cable operator Comcast, has ended up in the global carrier top 10. And it’s turned its uplink/downlink ratio around while doing it - Comcast is now a net sender to the global Internet.

In 2007, it was pulling into two or frequently even three bytes of data from the Internet for every byte it sent, the classic pattern of an eyeball network delivering content to read-heavy residential users. Now it’s a marginal net exporter of traffic. The exact figure helps to show what’s happened; Comcast’s traffic ratio is now hovering just a tad over 50%. That puts it in a group with the major transit providers; the big platforms send three or more bytes outbound for every one inbound.

That, in turn, suggests that their business is being driven by two-way communications applications - specifically, they’re providing mobile backhaul, metro-Ethernet connectivity for businesses, voice over IP transit (the quintessential 50/50 ratio application), and wholesale video delivery to other ISPs, like a CDN. Richer wholesale is driving their business model.

Remain Calm: Buy Data Centres

The overwhelming conclusion from this data is that the platform - for content delivery, for better wholesale, and for cloud computing - is where it’s at. Big platforms, we said, are the load-centre container ports of the digital economy, our future platforms for growth. As the Google engineers are in the habit of stickering on their laptops: My other computer is a data centre.

(thanks to licio)