« iPhone 4S take, Steve Jobs, US stats, new policy in India - Telco 2.0 News Review | Main | Voice & Messaging 2.0: Vodafone and DTAG to discuss RCSe ‘OTT defence play’ »

BlackBerryFail explained + how iOS/OSX updates melted networks - Telco 2.0 News Review

[Ed. It’s now just three weeks to the London Brainstorm (9th-10th November). Key themes are: strategies for defending and extending voice, Customer Experience 2.0, M-Commerce 2.0, Cloud 2.0, M2M 2.0, CDNs, Payments 2.0 and dealing with the disruptors - Google, Apple, Facebook, Microsoft/Skype, and Amazon.]

This week saw a major technology disruption as RIM’s BlackBerry service network failed for days in Europe, the Middle East, and Asia. Repairs were repeatedly promised. Then the outages spread to the Americas. Not only did it fail, it kept failing. It couldn’t have come at a worse moment. RIM, under pressure, has recently rolled out some impressive new devices like the Bold 9900 and Torch 9810, come up with good ideas like music-sharing via BBM, and made a good decision like hiring Alec Saunders.

But this can only be very bad news. The Observer reports that major British banks are seriously considering abandoning the BlackBerry platform and transitioning their mobile device fleets to Apple iOS. It’s not the first time, of course, that RIM has had a continental-scale outage. In fact, the highly centralised network architecture almost seems to invite disaster. Most of the world’s BlackBerry traffic passes through two pairs of data centres, in Waterloo, Ontario and in Slough and Egham in the UK.

That includes not just e-mail and BBM messaging traffic, but also all Internet traffic to or from a BlackBerry device, as RIM uses a bank of accelerator proxies in the cloud in a similar fashion to Opera Mini or Amazon Silk. For much of its existence, this has been a major factor in the BlackBerry user experience - when mobile networks were seriously bandwidth-constrained, anything that tended to speed up load times was worth having. Unfortunately, it’s also a massive single point of failure, and losing the BlackBerry Network Operations Centre means knocking all the devices back to the status of a 1990s GSM phone. Even on-premises BlackBerry Enterprise Server installations can’t function without the RIM NOC.

It seems that the NOCs are organised in an active-passive mode, with each pair consisting of a primary and a hot standby. In the UK, the primary is Slough and the backup is Egham. However, reports suggest that the backup NOCs haven’t scaled up quickly enough to keep up with the surge in data usage, and that when a hardware failure took out the Slough NOC, the failover site couldn’t cope. Eventually, a major database became corrupted, necessitating a complex restoration from backups.

This Daily Telegraph story points the finger at a Cisco Ethernet switch and an Oracle database, but the really interesting bit is that apparently RIM first found out about the crisis when UK MNO engineers noticed that BlackBerry traffic over their radio networks had fallen to zero and called the RIM NOC, which suggests RIM’s own network instrumentation leaves something to be desired.

Data Center Knowledge rounds up the fallout, pointing out that RIM may decide to create more data centres and decentralise the network, but this may mean more pressure from various governments to spy on their traffic. Reuters calls on the Slough NOC and reports that BlackBerry users were openly insulting RIM employees outside the building.

As a peace offering, RIM is going to give away a selection of premium apps, while further offering its enterprise customers a month’s free technical support. Some might suggest that the customers might do better offering RIM tech support.

They also released two new gadgets, but this won’t be at the top of anyone’s mind.

In Apple news, Tim Berners-Lee appreciates Steve Jobs and relates how he very narrowly missed a live demo of the very first Web implementation at a NeXT developer conference in France. Meanwhile, Horace looks into the costs of Apple Stores, and puts together a must-read post arguing that Apple’s secret sauce is quite simply that it’s a hell of an industrial company and that it’s quietly been investing heavily in tools and infrastructure that (presumably) its contract manufacturers operate. This is Tim Cook’s Apple, of course - the relatively unglamorous but enormously important world of the global supply chain.

Asymco’s analysis of Apple’s investment in infrastructure
Source: Asymco

Stupid Apple Rumours does the math and attempts to work out which Apple news/fansites are actually worth reading and which should simply be filtered. They conclude that only AppleInsider is worth reading, but this is no great recommendation - its rumour hit-rate is in line with that of Apple rumours in general, while the other major Apple blogs are actually less accurate than a random sampling of rumours would be.

It seems that Samsung’s president and COO has been invited to Steve Jobs’ funeral, where he will have a quiet meeting with Tim Cook over the bitter lawsuits between the two companies. It sounds like something out of The Godfather - will there be a dynastic marriage to cement whatever settlement they reach, or will they go to the mattresses in a war to the knife?

The two men who found a prototype iPhone 4 in a bar and sold it have been sentenced to a small fine and probation.

Apple pushed out the iOS 5 update this week, via Akamai’s CDN. Even Akamai couldn’t prevent many users having to wait ages for their update to download. PlusNet’s blog reports on the massive traffic spike on their Akamai cluster, plus an associated surge in iTunes downloads, although apparently putting so many iDevices and Macs out of action made a big dent in the BBC iPlayer streaming traffic. RevK at A&AISP reports on their experience. It is noted in the comments that Mac OS X was also updated, while Ubuntu Linux had a major release and the Steam online gaming platform pushed out a large patch. No wonder it was a night to remember in the NOC.

For their part, the CDN market leader pre-announced a major upgrade ahead of their imminent Edge 2011 customer conference. It looks like Akamai’s platform is going to see major changes in the areas of IPTV, media services and analytics, mobility, and integration with cloud computing systems. IBM and Riverbend Technologies are key partners. Details are expected at the conference later this week.

Last week we published research on CDNs 2.0: should telcos compete with Akamai? just as KDDI became the latest operator to buy a content-delivery network, CDNetworks. Skyfire’s new video accelerator product, Rocket 2.0, is out and claiming to squeeze down video streams for mobile networks.

Hulu’s owners give up on selling the video-streaming player. Verizon FiOS has a unique approach to customer retention. Vodafone and Virgin Media partner to snag a major deal for Ethernet, mobility, and unified communications.

The Guardian’s technology blog weighs into the UK 4G spectrum debate. Renesys reports that Internet latency in the Middle East is coming down steadily as more local interconnection comes on stream, and predicts that a new cable is going to transform Lebanon’s telecoms market.

Sony Ericsson, meanwhile, reported quite respectable results. The really interesting detail, though, is that they are planning to drop all other products except for their Android-based smartphones. This makes SE the first established vendor to join Apple and HTC as an all-smartphone player.

Even Nokia may have a reasonable quarter, Economic Times reports, thanks to dual-SIM devices in India and an inventory restock in China.

If you found Siri, the iPhone 4S virtual assistant, impressive, here’s Speaktoit, an Android application that implements something similar. AppMobi launches a new developer kit for PhoneGap and HTML5 cross-platform mobile apps. Ericsson adds in-app carrier billing to its IPX product line.

However smart your phone may be, one in six of them is contaminated with faecal bacteria.

France Telecom, in the form of Orange Business Services, has reached the start line for real M2M roaming - they are going to launch SIM cards using the 901 MCC, reserved for special M2M tasks, so that their customers can be their own mobile network while using Orange’s roaming relationships. We know someone who’ll be pleased.

In other deeply Telco 2.0 news, O2 UK is trialling an app that lets their iOS users make high-definition voice calls over O2’s (and indeed any) WLAN networks as a way of outflanking the OTT players.

BlueVia publishes a helpful tutorial on how to build Python applications using the BlueVia API.

With the European Union apparently happy, the Microsoft-Skype deal closed this week. Another Tropo tutorial. Remembering the blue box and AT&T’s effort to suppress the Esquire article on it.

In sadder news, this week, a giant of computing left us. Dennis Ritchie, known as dmr from the Bell Labs e-mail address he used on newsgroups and lists all over the Internet, joined Bell Labs in 1967 to work on the massive Multics project to develop an advanced time-sharing operating system. Multics ran aground in what became known as the “software crisis”, the historic realisation that managing very large software projects is inherently complex and risky, and was abandoned.

Ritchie and Ken Thompson, however, decided to keep going, cutting down the bloated codebase to a bare minimum. They were aided in this by the fact Bell Labs managers didn’t think the project was worth supporting, and as a result the only computer available had only 16KB of RAM. The result, which started up in early 1970, was of course UNIX, the operating system that basically underlies almost everything in computing. Ironically, it was the system’s command-line text editing tools that really impressed the other Bell Labs programmers.

For his next trick, Ritchie decided that the new OS needed a new programming language to make it genuinely cross-platform. As far back as 1964, researchers at Cambridge University and Imperial College London had been trying to invent something like this, but their Combined Programming Language (CPL) also became a victim of the software crisis. One of its inventors, Martin Richards, then moved to MIT, where he created a simpler, cutdown version of CPL called Basic CPL or B for short. Ritchie was inspired and made a large number of improvements to B, calling the result C. UNIX was then re-implemented in C, creating a package of OS, programming language, and developer tools that would theoretically run on any machine. It was a world-changing achievement. Ritchie later wrote the classic textbook The C Programming Language and stayed at Bell Labs for his entire career.

In regulatory news, EverythingEverywhere’s effort to slow down the sale of some spectrum is confirmed. It’s possible that they may try an “insurgent” launch of LTE in the band. It was the week the UK didn’t actually deploy a national porn filter or indeed change anything much.

The Times of India reports that the Indian Department of Telecommunications is not happy about MNOs with national roaming arrangements. RCR Wireless covers the new Indian national telecoms policy in some detail. It looks like they really mean it about national vendors.

Should the EU provide more spectrum for mesh networks and unlicensed working? PLDT is willing to hand some back as part of a takeover deal. The FCC demands evidence that AT&T really will onshore 5,000 call centre jobs if it gets to buy T-Mobile.

Verizon has changed its terms of service to let it collect and use a wide range of customer data for commercial purposes. Everyone is opted in by default, although you can opt out through the Account Tools webpage. Safaricom is going to deploy IMSI catchers to stop Kenyan prisoners using M-PESA in jail. Analysis of the German government trojan. It’s OK to track cellphones without a warrant in the US. Here’s the EFF’s advice on phones at demonstrations.

A fascinating story about working on Google Wave. After this epic Googler rant about getting it wrong with platforms and how Amazon got it right, it seems that Google’s API design competence is concentrated in the Australian group it bought for the technology behind Maps and that also did Wave. Meanwhile, Google Buzz, Jaiku, and Code Search are shut down. There’s an interesting discussion of Dart, Google’s new web programming language, here. How Google hires. And Google’s old offices, 165 University Avenue, Palo Alto, are now home to Shazam. The landlord buys a few shares in each startup that passes through just in case.

Microsoft discovers the joy of open-source software! Hadoop, the Google-inspired big data analytics system, is going to be an integral part of future Windows cloud products. ReadWriteWeb interviews Hadoop’s inventor about the move.

Amazon Web Services adds more private cloud features. You can now set up a Virtual Private Cloud and fill it with dedicated EC2 Spot Instances.

Cisco’s new VXI collaboration product line.

Should Mozilla fund Diaspora? Trying to fix British government IT with agile development.

In Digital Entertainment 2.0 news, Spotify is growing, but it’s still losing money. Does it really reduce piracy, increase it, or just have no impact at all? Netflix walks back its split from the DVD rental business, and the crowd goes wild. We’ll be writing more on this shortly, and sharing more at our special EMEA Digital Entertainment 2.0 Workshop on Tuesday 8th November in London.

And finally…

To share this article easily, please click:

Post a comment

(To prevent spam, all comments need to be approved by the Telco 2.0 team before appearing. Thanks for waiting.)

Telco 2.0 Strategy Report Out Now: Telco Strategy in the Cloud

Subscribe to this blog

To get blog posts delivered to your inbox, enter your email address:


How we respect your privacy

Subscribe via RSS

Telco 2.0™ Email Newsletter

The free Telco 2.0™ newsletter is published every second week. To subscribe, enter your email address:

Telco 2.0™ is produced by: