It’s Bad Enough

I know what people think. I know they think we just go out of our way to make these things look bad. Things at the FAA just can’t be as bad as we make them out to be. You can tell yourself that -- but it doesn’t change reality.

Reality check: I’m just an old, retired controller (still young to the real retired world.) I get paid nothing to do this. I don’t work for anybody or represent any organization. I don’t even have any ambitions. There’s no job I’m angling for -- no position. Even if there was, they couldn’t pay me whatever it would take to get me to move. (I’m told every man has his price but I haven’t found mine yet.) I’d be the first to tell them -- I’m not worth that much.

I do feel a sense of duty to my country. And I do feel an obligation to the people that send me a retirement check every month -- the U.S. taxpayers.

Without further ado, here’s what fell into my lap this morning.

The Story Behind the FAA Flight-Plan System Crash

”When the router went offline, only the system maintainer—government telecommunications contractor Harris—knew that the backup card was not immediately available, and that one technician, who hadn't come to work yet that day, had the key to the storage closet where the part was kept.

So the FAA had to wait until this technician was able to come to the site in Salt Lake City to replace the faulty card inside the router, reconfigure the software, and get the communications backbone back up and running so that the nation's air traffic could get back to normal. “

(Emphasis added)

Let me pause here to make sure everyone fully understands this. At this 24 hours a day, 7 days a week, 365 days a year safety-critical government facility -- a place that has the ability to all but shut down the nation’s entire aviation system -- the key to the spare parts closet is in the possession of a private contractor that isn’t even present at the facility.

It’s bad enough. I can’t make that look any worse than it is.

”Before 2002, when the FAA contracted out the FTI system to Harris, the system was maintained by FAA telecom technicians on duty 24 hours per day.“

In other words, before 2002, this problem (if it had occurred at all) would have been fixed in minutes (instead of 4 hours) by the technicians that staffed the facility 24/7/365. And the key to the parts locker would have been at the facility instead of roaming around the countryside.

Okay, now it’s time to delve into the inside baseball. For a system that is so critical, where is the system redundancy ? There are two facilities (Salt Lake and Atlanta/Hampton) to handle the load. The design is for them to back each other up. One facility can handle the entire load should the other fail. What happened ?

”The FAA utilizes the NADIN (National Airspace Data Interchange Network) communications link for the flight-plan system. The two NADIN sites in Salt Lake City and Hampton, Ga.—along with including the 21 other FAA IT stations—no longer use a multipath communications backbone composed of many different redundant links.

As mandated by the Bush administration in 2001, all the communications links that previously were government-owned and maintained by FAA employees were contracted to Harris, under the $2.4 billion FTI contract. “

I’ll repeat myself -- It’s bad enough. I can’t make that look any worse than it is.

”...no longer use a multipath communications backbone composed of many different redundant links.“

We (and I do mean we) told you so.

Don Brown
November 21, 2009

Share