E-Paper
How one router flooding the FAA’s network forced a nationwide US ground stop
A faulty router flooded the FAA's network with connection requests, disrupting radar feeds and aircraft communications and forcing a nationwide US ground stop.
Prefer HTon Google
A single router installed during routine maintenance may have triggered the ground stop that halted departures across the US for more than four hours on November 13, US officials have said, after it flooded other routers in the Federal Aviation Administration’s (FAA) communications network with connection requests.
The router was installed by government contractor L3Harris, which was replacing ageing copper lines with fibre-optic cable at a remote FAA site, Sean Duffy, the US transportation secretary, told reporters on November 14. “We had over 1,000 connection attempts every four to five seconds, and what that did was (it) gave us issues with our communication with airplanes, and it also caused some disruption with our radar,” Duffy said on .
Bryan Bedford, who heads the FAA, said the network did not fail outright. “No facilities went dark. Nothing was fully lost,” he said.
What does a router do in air traffic control?
A router is a networking device that directs digital information, broken into units known as data packets, between devices and networks. It helps ensure that information sent from one point reaches the correct destination.
In an air traffic control network, routers move information between control centres and airport towers. This includes data used for communications between controllers and pilots, and information from the radar systems controllers use to monitor aircraft.
A router does not detect aircraft or track their movements. It moves the information that other systems generate.
How did the router overload FAA’s network?
The incident occurred while L3Harris was carrying out routine maintenance to replace ageing copper telecommunications lines with fibre-optic connections at a remote FAA location.
According to Duffy, the work involved shifting from older time-division multiplexing (TDM) technology, which gives each call or data stream its own fixed slot on a line, to internet protocol (IP)-based communications, which break data into packets that share the network.
“Either through contractor error or a faulty piece of equipment, that router that was being replaced as we switched from copper to fiber, it started flooding other routers in our air traffic control system,” Duffy said, according to a DoT transcript of the briefing.
A router that receives connection requests at such a rate may have to devote much of its processing power and network capacity to handling them. That can congest the network and delay legitimate traffic between connected systems.
The exact technical fault has not yet been established. “We’re working to determine whether we had incompetence, meaning was it user error or installer error from L3Harris, or did they install a faulty router? We haven't determined what the fault was,” Duffy said.
L3Harris said it took the issue seriously and was supporting the FAA's investigation, Reuters reported.
Why would this affect communication with aircraft and radar feeds?
Controllers need to communicate with pilots and receive reliable information from radar systems to monitor aircraft positions and coordinate their movements. If the telecommunications network moving that information becomes unstable, data may be delayed or interrupted.
The FAA said its communications did not completely go down; the problem caused instability in data at its facilities. It affected communications at 17 of the FAA’s 22 en route centres, which handle aircraft during the cruise portion of a flight.
The FAA's concern was whether the information controllers needed to manage air traffic could still be relied on.
Why did the FAA ground flights across the US?
The FAA issued the nationwide ground stop, which holds departing flights at their airports, because it could not identify a root cause of the instability. Bedford said the agency ordered it in case one of the centres lost all communications.
In aviation, controllers need dependable information to separate aircraft and coordinate traffic safely. If the systems handling critical communications and radar data become unstable, continuing normal operations can introduce risks.
According to the DoT, more than 3,500 aircraft that were in the air were safely brought to the ground during the disruption.
The FAA lifted the stop at 8.15pm Eastern Time, and more than 8,000 flight operations resumed that night, the department said.
Asked whether the outage was a cyberattack, Duffy said: “We don’t believe so.” The investigation into the equipment and installation issues is continuing.
Also read: US flights to return to normal by Sunday after nationwide ground stop, FAA says
Ageing technology
The incident has also drawn attention to the challenges of modernising the US air traffic control system while it remains in operation.
The FAA has been replacing its decades-old copper telecommunications infrastructure with fibre-optic connections. Fibre uses pulses of light to transmit information and can move large volumes of data over long distances.
Duffy said the FAA had replaced 69% of its copper communications with fibre. Bedford said the agency spends about $1 billion a year maintaining its existing telecoms network. He described the work at the remote site as a maintenance upgrade at a single facility, separate from the $12.5 billion modernisation programme Congress has funded, according to the DoT’s account of the briefing.
This is not the first time a relatively small technical problem has halted flights nationwide. In January 2023, contract staff unintentionally deleted files while working on the database behind the FAA’s system for sending safety notices to pilots, the agency said. The resulting nationwide ground stop lasted about 90 minutes and disrupted more than 11,000 flights.
These upgrades must take place without interrupting a network that supports air traffic operations across the country. The November 13 incident has raised questions about how equipment replacements are tested and monitored, particularly where older and newer systems have to operate alongside each other.
