What should have been a run-of-the-mill software upgrade by
General Instrument Corp. crashed part of Comcast Corp.'s hometown system, taking
thousands of customers off-line, some for as many as four days.
The software glitch affected 70,000 customers in one
headend in Philadelphia that included the communities of Chestnut Hill, Germantown,
Hunting Park, Logan, Mount Airy, North Philadelphia and Olney. It was an embarrassment in
a town where Comcast has positioned itself as a telecommunications and sports-programming
GI stressed that although the software upgrade has been
used elsewhere in the industry, several conditions occurred at once in Philadelphia, which
made that situation unique.
"We've taken steps to keep this from happening
again," said Dick Badler, vice president of communications for GI. "It was one
of those things."
"One of those things" was a relatively common
multicontroller upgrade. But when the technician loaded it, it force-tuned
several-thousand boxes served by the headend to a pay-per-view barker and locked them
Murphy's law was at work: The outage began midday
Friday, March 27, the eve of the heavily viewed Final Four of the NCAA basketball
Michael Doyle, Comcast's regional senior vice
president, said the system still has not determined exactly how many of the 70,000 boxes
were blocked. However, the system quickly realized, based on the volume of calls, that the
outage was widespread.
"We brought in every employee we had," Doyle
The operator determined almost immediately that it only had
one channel to work with, so Comcast put a crawl on it, directing consumers to call a
special 800 number to have their boxes readdressed. Following that direction, the operator
was able to reauthorize, one at a time, 27,000 boxes within the first 36 hours. The
operator took e-mail from customers, giving them another line of communication, because
the telephones were busy.
But the crash illustrates the difficulty that the business
has in getting its messages across. Even with on-screen notification of the problem, a few
consumers complained to the local press that they didn't know what was happening.
Also, consumers care about their own needs (for example, whether their signal would be
restored in time for USA Network's Tuesday Night Fights), and there was no way
that the system could address those individual questions because it did not know how long
the outage would last.
Comcast reprogrammed the one channel that it had to work
with, so the affected homes still got to see the climax of March Madness.
Computer programmers from Comcast worked side-by-side with
GI coders during the weekend, and they developed a program that sped reauthorization by a
factor of 20. By midday Monday, the system was fully functional.
Doyle said GI president Ed Breen was in constant contact
through the weekend.
"I couldn't have asked for more from a company
... GI did not hide from the problem. It proves that if you hire a reputable vendor, they
will be there to fix the problem," Doyle said.
The operator notified city regulators promptly of the
problem and its efforts to fix it, he added. Comcast will give every affected customer a
credit, whether they were out for 10 minutes or for the duration of the crash, he said.
"I think that we did a lot of right things,"
A fast response was even more important because
Philadelphia is Comcast's headquarters city, and the company has a big investment in
state-of-the-art plant and community goodwill.
City regulators did not return repeated calls to comment on
Comcast's recovery actions.
Comcast executives will sit down this week and determine
the cost of the outage, which will include the credits and a weekend of overtime.
When asked whether he thought that the computer outage
would dissuade consumers from considering digital, which Comcast will launch soon, Doyle
replied flatly, "No."
Consumers were notified of the credit offer by letter April