How Art Changes the Architecture of the Internet

Reading Neelie Kroes’ recent blog post on net neutrality (link updated Dec 2017) reminded me of a startling presentation that was given to Nanog in 2009 (I was not there so was thankful that the slides were made available here) which brought into focus some vague ideas I have been thinking about for the last few years.

That 2009 report was one of those ‘everything you thought you knew about x is wrong’ moments. Because by 2009 the open, public, internet, with a predictable hierarchical cascade from core to access, was over. Instead we had arrived at a world of private connections between big content and service platforms where IP protocols carried private bits over private networks. And what drove the change was, bizarrely, art – or rather content if you want to avoid aesthetics. More specifically, Google’s YouTube acquisition could create so much traffic for ISPs that it was uneconomic to carry it via IP transit, bought from upstream providers. Google had to be let into the peering points, and the biggest ISPs choose to peer privately with large partners to avoid overloading the public switches and getting into complicated recharge discussions.

Earlier in the evolution of the internet CDNs had been let into the exchanges, to some grumbling, but as wholesale suppliers of connectivity they sit behind the consumer services so arguably were similar in their role to the wholesale transit providers. Having a consumer service in the networks club was a new experience.

Now here’s some conjecture and speculation. Carpathia, a Dulles, Virginia blue chip hosting business, joined a select club when it accepted a new customer, MegaUpload, in November 2008. Almost instantly it was generating 1/20th of all internet traffic. If Carpathia did not manage to use that leverage to move into the internet exchanges and form some private peering agreements it should have. Private traffic is cheaper and inherently less visible than public. And some more speculation on a historical matter – the apparent drop in internet traffic resulting from Sweden’s introduction of IPRED laws could at least partly have been the effect of some publicly peered traffic going private for similar reasons.

A minor version of this happened very recently when customers of UK ISP Virgin Media were reporting severe difficulty using Spotify. The Register extracted a comment from Virgin Media to the effect that they had been suffering from poorly configured peering with one of Spotify’s data centres. To fix it they will have either turned up the public peering, or gone private. Either way the shape of the internet changed in response to demand for music.

I suspect it would be possible now to avoid the internet entirely and just use private networks if your access comes from a major consumer ISP. And it is art/content that has driven this change. How much the content creators and their industrial representatives have been party to the changes is anyone’s guess, but I suspect not very much. That looks like an opportunity to me.