Ops Dilemma: Get More Out of Pipes

Publish date:
Updated on

Road Runner and At Home Corp. are hustling to find ways to
overcome problems in cable operator architecture in delivering high-speed data even as
MSOs' efforts in this area are facing severe new challenges.

In a nutshell, the problem technical strategists face is
how to make smart pipes with limited, shared capacity act like very fat dumb pipes as the
number of customers industry-wide surges at a rate of 10,000 or more per week.

"Some operators don't realize it, but they're in the
software support business now," said Milo Medin, chief technical officer for At Home.
"This isn't just about delivering communications services."

Translated, this means MSOs and their affiliated service
providers must be able not only to use sophisticated software to efficiently manage
bandwidth. They must also be very good at communicating to subscribers the nuances of how
the myriad personal software options now on the market can interact with network software.

In some cases the software can change a given user's
experience of how the carefully orchestrated cable data network performs.

And cable data service providers have to be even better at
troubleshooting problems by asking the right questions about those personal software
choices when a dissatisfied customer is on the line to the help desk.

"A customer can be inadvertently turning off the
caching [local storage and distribution mechanism] because he's downloaded some piece of
shareware," Medin said.

Or a customer can download a streaming media software
"player" from Microsoft Corp., for example, that overrides the player that works
with streamed media from one of @Home's content partners, resulting in poor performance
from content that is supposed to be "broadband enhanced."

A new version of @Home's end-user software is designed to
address such problems by periodically monitoring and resetting the user's commands to
optmize performance. But the balancing act will still require good, knowledgeable
communications with end users when they have problems, not to mention existing users'
willingness to download the new @Home software, Medin said.

Newly installed customers will be given the fresh software
from the outset, he noted.



Ironically, one of the biggest challenges in getting
subscribers to properly use network intelligence involves the sophisticated early adopters
who comprise the core of customers who have been taking services the longest.

As has been the case recently in @Home service areas of
southern New England, these advanced customers often discover that their usual approaches
to using data networks can wreak havoc with cable data service as the number of users

As previously reported, customer outrage over slow data
transfer rates and unresponsive customer service in Connecticut and Rhode Island brought
things to a head over two weeks ago. That's when a key user group scheduled a
demonstration against @Home in Bristol, Conn., where Tele-Communications Inc.'s TCI.NET
unit was demonstrating the service at an automobile dealership.

But TCI and @Home successfully defused the situation when
top officials visited the site on Feb. 27 and turned it into technical seminar, according
to Scott Greczkowski, the leader of the user group.

The user group allowed the TCI marketing event to proceed
successfully with the signing of over 600 new customers.



"As of the weekend of the demonstration, the network
was working flawlessly," Greczkowski said. "We were happy that @Home and TCI
sent the representatives to talk to us and didn't see any reason to be disruptive."

However, he added, as of the following Monday and
continuing through the week, the users were finding the service was slowing down again,
dropping on occasion to dialup data rates and sporadically freezing up.

"I'm convinced they're making a sincere effort to
correct the problems," Greczkowski said, "but some people are wondering why
everything worked when the situation came to a head and suddenly goes back to being poor
after the publicity crisis is over."

In fact, the volatility of network performance is a
function of a complex balancing act involving many technical elements that can be
extremely hard to control in situations where steps haven't been taken in advance to
accommodate surging subscriber growth.

Big changes are coming, @Home's Connecticut customers were
told, but, for now, there are no guarantees that things will run smoothly.

In New England, @Home is adding higher capacity links
between the company's regional data center and its national backbone switches and tying
the Rhode Island market directly to the backbone, But while @Home is taking some steps
specific to its New England markets, most are focused on changes that will affect all

In effect, the goal is to apply more sophisticated tools
and better customer service support in the interest of making growth less painful.

Both @Home and the Road Runner Group, the other leading
cable data service provider, are in the midst of major initiatives to accommodate their
growth. The Cable Broadband Forum estimates their combined customer bases have risen by
100,000 since the start of the year to over 600,000.

Time Warner Cable and MediaOne Group Inc. are the major
stakeholders in Road Runner; @Home is jointly owned by TCI, Comcast Corp. and Cox
Communications Inc.



While both entities are addressing the same types of
issues, their approaches are often quite different.

For example, @Home has contracted with AT&T Corp. to
lease capacity for a broadband backbone network to be turned on starting next month, in
effect making it a national transport provider. Road Runner is focusing on building
regional high-capacity ring networks while relying on tie-ins with the national IP
backbone operated by Qwest Communications. (See story, page 43.)

Medin said the national @Home network will make it much
easier to "peer" or interconnect with other major data transport providers. It
uses a new combination of router and optical technology that allows direct interconnection
with other providers within shared "facilities hotels" at major carrier points
of presence. This avoids the need to secure use of a third party's local links to make
such connections when the facilities are widely separated.

"Now we'll be able to physically run fiber from our
facilities to those of another provider, which eliminates the whole process of ordering
lines from other carriers," Medin said.

Even with the new backbone, there are problems with local
trunking capacity to be addressed in other areas besides Connecticut, Medin acknowledged.

For example, the squeeze on service performance from rapid
subscriber growth was showing up in performance complaints from second tier markets such
as Birmingham, Ala.; Portland, Ore.;, and Nashville, Tenn. These, like other recent @Home
problems, quickly caught press attention, this time in a report by the San Francisco

"As we turn the new backbone on we'll be adding
trunking capacity in the smaller markets," Medin said.


At Road Runner, managers are putting in place a series of
high-speed optical rings linked to regional data centers that provide a number of
connecting points for local systems, eliminating the separate trunking feeds into the RDC,
said Steve Zan Beaver, senior vice president for operations.

The first such OC-48 (2.5 gigabits per second) ring went
into operation in New York state last year, linking systems in the Birmingham, Syracuse
and Rochester areas.

"So far, we're holding our own," Zan Beaver said,
in reference to avoiding major service problems. The key to doing so, he added, is to
build a "fudge factor" into projected subcriber estimates so that the network
stays ahead of the growth curve.

Proactive steps underway at Road Runner include: ordering
of DS3 (45 megabits per second) trunks from transport vendors well ahead of anticipated
needs for such capacity; expansion of system server capacities ahead of growth estimates;
and creation of a network operations center near the company's new headquarters in
Herndon, Va.

The new NOC, now in temporary quarters, provides operations
managers monitoring information covering all affiliates to the headends, instantly
signaling trouble spots at any point along the way, Zan Beaver said.

Zan Beaver stressed that while Road Runner faces many of
the same issues @Home does, it has not had the same volume of customer complaints to deal
with. That assertion is seemingly confirmed by the recent evidence of online complaints
circulating over the Web and the local and national press reports that have largely
focused on @Home.

In part, Zan Beaver said, this difference reflects the
aggressive pro-active steps Road Runner has taken and in part it reflects that @Home is
serving many more MSOs and, therefore, a wider range of local networking environments and
operating procedures.

Speaking privately, an @Home official acknowledged that it
is much more difficult dealing with the operating conditions at some cable companies than
at others.

"You look at Comcast, Cox, for example, we're just not
having the problems with them that we're having with some of the other companies," he

But no matter what the local operating situation might be,
it's clear that the cable industry is going to have to be aggressive in staying ahead of
the rapid growth of its data subscriber base. In an environment where the tools in
subscribers hands together with the types of content and applications offered by
innovators over the Web are constantly changing, the scaling challenge only gets more
multidimensional over time.