Bug Blog

Problems in Denver

In Denver a software glitch left at least one person on hold for five minutes.  The glitch was caused by calls that had already been dealt with being looped back into the system and reported as hang up calls.

911 operators are required to call hangup numbers back as standard procedure, in order to insure that the caller is safe.  This ties up the phone lines causing people who have emergencies to wait on hold.  Calls can be repeatedly looped back into the system causing a tangle that is hard to unwind and locks up the system.  Calls coming in over T Mobile lines were looped through again and again, causing wasted resources and lost time.

T Mobile has been working with Denver authorities to determine the cause of the problem and the number of looped calls has been reduced, but not yet eliminated.

T Mobile issued this statement regarding the issue, " We take our customers security and safety very seriously.  We previously identified an isolated issue and, in collaboration with Denver 911, addressed it.  We will continue working closely with Denver 911 and monitor the situation." Denver authorities want to resolve this issue quickly before the summer months when call volume greatly increases.  Denver authorities have also requested that people not hang up when calling 911 as this only delays their getting help.  It takes longer for 911 operators to call them back than the amount of time they would have to wait on hold.

Denver 911 Executive Director Athena Butler said that the software glitch was a serious development, "It can be very serious and frustrating because while we are calling someone back who hasn't called us, someone else is calling in who needs help."

And Butler expressed how looping calls can be a problem. "We have had some looping calls come in and it took us about five minutes to call someone back. They were on the phone for three minutes, they hung up. We were on other calls, after about two minutes, it dropped to somebody to call them back. That's how busy we were during that time period."

Butler also advised callers not to hang up, even if on hold as it actually takes longer for operators to call them back than the time spent on hold.

The Denver problem began in 2015, but only really came to notice about a year later when operators began keeping track of looped calls. This may seem like a long time, but the cause is not neglect on the part of Denver authorities. The sheer complexity of modern software means that bugs can hide just below the surface and only rise to notice when they become acute. This is another reason for periodic testing in the field, especially in high risk operations where failure could cost lives or a large amount of money.

Problems in Dallas

Back in October of 2016 Dallas 911 began to notice that it was receiving multiple calls from the same numbers. It was as if Mobile phones used to call the emergency number were dialing 911 over and over again. At first, the problem was intermittent and didn't increase hold times. However, the problem grew considerably over the intervening months.

As it turns out, the ghost call glitch in Dallas was not T Mobile's fault.  The problem was caused by wait times that were too long.  People would hang up before the phone was answered causing the " ghost calls" and tying up the lines as 911 operators tried to call them back.  And many people could not be reached because they were already on hold calling 911 back.  This is a problem that can be easily solved by adding a few technological upgrades and more 911 operators.

City Manager T.C. Broadnax said “T-Mobile committed resources in Dallas until we made progress, and they have kept their promise. We want our citizens to know that their safety is our number one priority and they can count on us when they call 911".

And Mayor Mike Rawlings added, I'm pleased that our staff and T-Mobile worked through the night and have determined some immediate technological upgrades that will better serve our citizens calling 911. I'm also encouraged that City Manager T.C. Broadnax has committed to increasing staffing and recommending any other budgetary enhancements to our 911 call center.”

T Mobile engineers have made improvements to the Dallas 911 Network and are keeping an eye on the situation to make sure that no new issues arise.  They are assisted by third party vendors who are part of the support team that maintains the public safety infrastructure for Dallas.

It's really just a number

The 911 emergency phone line is really just an ordinary phone number that primarily uses the same infrastructure as the rest of the phone network. While the receiving end of 911 calls may receive more attention in terms of maintenance and serviceability, it is subject to the same problems that beset any other phone number.

We've all had phone connectivity problems at one time or another and 911 is no different. It still requires a working telephone and functioning cables and/or cell towers.  It also requires a call center, called a Public Safety Access Point, that can properly route calls. Different areas are served by different call centers and these may all be linked through a single control point. Software is required all along the way, from the telephone of the person reporting an emergency all the way through to the dispatching of emergency response. Actually, it works pretty good most of the time, when you consider the complexity involved.

If the T Mobile bug and problems in Dallas show anything, it is that as our society becomes ever more dependent on digital technology, the requirement to test software both before and after deployment and to continuously maintain and update infrastructure, will only increase.  The need for careful planning and the rigorous testing of software, has now become the norm.