Update: last week's FAA system outage was caused by the "unintentional" deleting of files from the NOTAM system

CitizenVectron · January 11, 2023

Flights across the U.S. grounded after FAA experiences computer outage

WWW.NBCNEWS.COM

The Federal Aviation Administration said it had "ordered airlines to pause all domestic departures until 9 a.m. Eastern Time to allow the agency to validate the integrity of flight and safety information."

Quote

Flights across the United States were grounded Wednesday morning after the Federal Aviation Administration said it experienced a computer outage.

All flights in the U.S. were grounded following the incident, a source with knowledge of the situation told NBC News. The FAA said later Wednesday morning it had ordered all airlines to pause domestic departures until 9 a.m. ET.

Quote

More than 1,200 flights within, into and out of the U.S. were delayed as of around 7 a.m. Wednesday morning, according to online flight tracker FlightAware. Just over 100 flights were listed as cancelled.

Quote

The FAA had said in a notice on its website that its Notice to Air Missions (NOTAM) system had “failed.” A NOTAM is a notice containing information essential to workers involved in flight operations.

best3444 · January 11, 2023

**Commissar SFLUFAN** · January 11, 2023

The ground stop is in place until 9:30am Eastern. The systems are coming back online.

January 11, 2023

NOTAM and similar attacks have to be a pretty high priority for entities wishing to create chaos in the US. I’m flying out of LAX shortly and my flight is delayed from the knock-on effects of the earlier grounding. Not difficult to imagine how much of a mess you could create with the system down for 12+ hours.

unogueen · January 11, 2023

Yer security is arrogant. 15 year old me knew this. Not getting fixed until you give a shit.

**Commissar SFLUFAN** · January 20, 2023

gettyimages-1455793055_wide-15a19aa02354

FAA contractors deleted files — and inadvertently grounded thousands of flights

WWW.NPR.ORG

The agency's preliminary review determined it was unintentional as the contractors worked "to correct synchronization" between databases.

Quote

Contractors unintentionally grounded thousands of flights last week when they deleted files while working on the Notice to Air Missions (NOTAM) system, the Federal Aviation Administration says.

The agency said in a statement Thursday that a preliminary review found the shutdown happened as the contractors worked to "correct synchronization between the live primary database and a backup database." Investigators so far found no evidence of malicious intent or a cyberattack.

NOTAM is used by the FAA to notify pilots and airports of any potential flight hazards.

Keyser_Soze · January 20, 2023

LazyPiranha · January 20, 2023

Subzwari1987 · January 20, 2023

Giving contractors read-write permission to your critical production servers :tired:

CayceG · January 20, 2023

If I were the project manager, I would be pursuing liquidated damages so fast.

SuperSpreader · January 20, 2023

We've all done it

SuperSpreader · January 20, 2023

Now hear me now, there's a concept called backups

SuperSpreader · January 20, 2023

Try FAA_systemfinal838483-final-463b_v2

Subzwari1987 · January 20, 2023

15 minutes ago, SuperSpreader said:

Try FAA_systemfinal838483-final-463b_v2

Wouldn’t surprise me if they store the day’s airport schedule in an excel file named JFK_Map_2022-01-15_v5

Keyser_Soze · January 20, 2023

FAA_File_DontDelete

b_m_b_m_b_m · January 20, 2023

28 minutes ago, SuperSpreader said:

Now hear me now, there's a concept called backups

And testing!

Subzwari1987 · January 20, 2023

6 minutes ago, b_m_b_m_b_m said:

And testing!

Pfft. Real men test in Prod.

SuperSpreader · January 20, 2023

11 minutes ago, Subzwari1987 said:

Pfft. Real men test in Prod.

I can't make out if he's doing nerd talk or wheezing! It sounds the same to me!

CitizenVectron · January 20, 2023

Yeah, when the news first broke that this was a single person's fault, my first thought was "well...is it really a single person's fault if the system allows a single person to just stop all flights?" There should be a huge change request process in place that implements in dev environments, gets multiple people to sign off, etc.

I hate those processes as much as anyone...but in systems that are large as all airlines in the US, they are necessary.

**GeneticBlueprint** · January 20, 2023

Red tape isn't always (and most often is not) bad.

b_m_b_m_b_m · January 20, 2023

It makes work more predictable and less likely to blow up which is fine and very good if you’re not getting actual equity from the job, at least everywhere I’ve worked this is true at least. And the one time the process wasn’t followed it caused an issue which required reporting to the government and was a year+ long multimillion dollar mess

Link200 · January 22, 2023

On 1/20/2023 at 10:58 AM, SuperSpreader said:

Now hear me now, there's a concept called backups

The were probably on tape drives from 1969.

This issue was annoying I must say. Took me all day to commute home. That said, NOTAMs need completely changed. Maybe this will kick them in the butt to change them.

BTW these are some examples of NOTAMs.

Spoiler

KDFW
2/0081 DFW IAP DALLAS-FORT WORTH INTL, DALLAS-FORT WORTH, TX. RNAV
(GPS) RWY 36R, AMDT 3C... LNAV MDA 1020/HAT 439 ALL CATS. VIS CATS C/D
RVR 4000. FOR INOPERATIVE MALSR, INCREASE LNAV CATS C/D VISIBILITY TO 1
1/4 SM. VDP NA. TEMPORARY CRANES, 707 MSL, BEGINNING 1.32 NM SOUTH OF
APCH END RWY 36R (2020-ASW-4175 THRU 4178-NRA). 2212051610-2307171610

KDFW 12/655 DFW TWY K BTN TERMINAL C RAMP SPOT 38 AND TERMINAL C RAMP SPOT 43 CLSD 2212191200-2305012359

KDFW 12/706 DFW TWY R5 CL MARKINGS BTN APCH END RWY 31R AND TWY R NOT STD 2212201909-2301312359

KDFW 01/576 DFW RWY 18R/36L CL MARKINGS OBSC 2301201405-2302202359

KDFW 01/104 DFW TWY G BTN TERMINAL B RAMP SPOT 122 AND TERMINAL D RAMP SPOT 124 WIP STEEL PLATES 2301050440-2303312359

KDFW
2/6506 DFW IAP DALLAS-FORT WORTH INTL, DALLAS-FORT WORTH, TX. RNAV
(GPS) RWY 17C, AMDT 2B... LNAV MDA 1140/HAT 578 ALL CATS. VIS CATS C/D 1
1/4. VDP 1.58 NM TO RW17C. FOR INOPERATIVE ALS, INCREASE LNAV CATS C/D
VISIBILITY TO 1 5/8 SM. TEMPORARY CRANE, 757 MSL, 1.47 NM NORTH OF APCH
END RWY 17C (2019-ASW-15513-OE), TEMPORARY CRANE, 822 MSL, 3.33 NM NORTH
OF APCH END RWY 17C (2022-ASW-15229-OE). 2211031351-2306151351

KDFW
3/3397 DFW SID DALLAS-FORT WORTH INTL, DALLAS-FORT WORTH, TX. AKUNA
NINE DEPARTURE (RNAV)... ALIAN TWO DEPARTURE (RNAV)... ARDIA SEVEN
DEPARTURE (RNAV)... BLECO EIGHT DEPARTURE (RNAV)... COYOTE THREE
DEPARTURE... DALLAS TWO DEPARTURE... DARTZ NINE DEPARTURE (RNAV)...
FORCK THREE DEPARTURE (RNAV)... GARLAND FOUR DEPARTURE... GRABE EIGHT
DEPARTURE (RNAV)... HRPER THREE DEPARTURE (RNAV)... HUDAD TWO DEPARTURE
(RNAV)... JACKY NINE DEPARTURE... JASPA SEVEN DEPARTURE (RNAV)... JOE
POOL SIX DEPARTURE... KATZZ TWO DEPARTURE (RNAV)... KEENE FOUR
DEPARTURE... KINGDOM THREE DEPARTURE... KUSSO ONE DEPARTURE (RNAV)...
LOWGN EIGHT DEPARTURE (RNAV)... MRSSH THREE DEPARTURE (RNAV)... NELYN
SIX DEPARTURE (RNAV)... SWABR ONE DEPARTURE (RNAV)... TEXOMA FOUR
DEPARTURE... TRI-GATE FOUR DEPARTURE... TRYTN FOUR DEPARTURE (RNAV)...
WORTH NINE DEPARTURE... WSTEX TWO DEPARTURE (RNAV)... ZACHH FOUR
DEPARTURE (RNAV)... TAKE-OFF MINIMUMS RWY 13L, 300-2 1/4 OR STANDARD
WITH MINIMUM CLIMB OF 247 FT PER NM TO 1000, OR ALTERNATIVELY, WITH
STANDARD TAKEOFF MINIMUMS AND A NORMAL 200 FT PER NM CLIMB GRADIENT,
TAKEOFF MUST OCCUR NO LATER THAN 1400 FT PRIOR TO DER, TEMPORARY CRANE,
875 MSL, 5691 FT EAST OF DER RWY 13L (2022-ASW-3544-OE). RWY 17C, 300-1
OR STANDARD WITH MINIMUM CLIMB OF 280 FT PER NM TO 800. RWY 35R, 300-1
3/4 OR STANDARD WITH MINIMUM CLIMB OF 352 FT PER NM TO 1000 . TAKEOFF
OBSTACLE NOTES: RWY 17C, TEMPORARY CRANES, BEGINNING 4978 FT FROM DER,
BEGINNING 1555 FT LEFT OF CENTERLINE, 180 AGL/737 MSL (2020-ASW-6805,
6809, 6810-NRA), RWY 31R, TEMPORARY OIL RIG, 5378 FT FROM DER, 1431 FT
LEFT OF CENTERLINE, 115 AGL/696 MSL (2019-ASW-142-NRA). ADD TAKEOFF
OBSTACLE NOTE: RWY 17R, TEMPORARY CRANE 4178FT FROM DER, 1224FT RIGHT OF
CENTERLINE, 140FT AGL/ 694FT MSL (2021-ASW-2327-NRA), RWY 35C,
TEMPORARY CRANE, 3574 FT FROM DER, 572 FT RIGHT OF CENTERLINE, 105
AGL/656 MSL (2022-ASW-4281-OE). ALL OTHER DATA REMAINS AS PUBLISHED.
2301131238-2308251238

KDFW 2/6621 DFW IAP DALLAS-FORT WORTH INTL,
DALLAS-FORT WORTH, TX. RNAV (GPS) Y RWY 31R, AMDT 3A... LP MDA 1120/HAT
597 ALL CATS. VIS CATS C/D 1 1/4. LNAV MDA 1160/HAT 637 ALL CATS. VIS
CATS C/D 1 3/8. FOR INOPERATIVE ALS, INCREASE LP AND LNAV CATS C/D
VISIBILITY TO 1 3/4. DISREGARD GUKSE FIX AND GUKSE FIX ALTITUDE.
TEMPORARY CRANES, 684MSL, BEGINNING 1.30 NM SE OF APCH END RWY 31R
(2019-ASW-2180 THRU 2183-NRA). TEMPORARY CRANES, 679 MSL, BEGINNING 1.30
NM SE OF APCH END RWY 31R (2020-ASW-6897/6898/6901-NRA), TEMPORARY
CRANE, 875 MSL, 5691 FT EAST OF APCH END RWY 31R (2022-ASW-3544-OE).
2209131610-2304251610

KDFW 2/3637 DFW IAP DALLAS-FORT WORTH INTL,
DALLAS-FORT WORTH, TX. ILS OR LOC RWY 36L, AMDT 4... S-LOC 36L MDA
1120/HAT 532 ALL CATS. VIS CATS C/D RVR 5500. SIDESTEP 36R MDA 1120/HAT
539. VDP NA. CIRIS FIX MINIMUMS (DME REQUIRED): DISREGARD CIRIS FIX,
CIRIS FIX ALTITUDE AND CIRIS FIX MINIMUMS. TEMPORARY CRANE, 804 MSL,
3.02 NM SOUTH OF APCH END RWY 36L (2021-ASW-6363-NRA), TEMPORARY CRANES,
707 MSL, BEGINNING 1.34 NM SOUTH OF APCH END RWY 36L (2020-ASW-4175
THRU 4178-NRA). 2206241352-2302031352

KDFW 01/321 DFW AD AP RDO
ALTIMETER UNREL. AUTOLAND, HUD TO TOUCHDOWN, ENHANCED FLT VISION SYSTEMS
TO TOUCHDOWN, HEL OPS REQUIRING RDO ALTIMETER DATA TO INCLUDE HOVER
AUTOPILOT MODES AND CAT A/B/PERFORMANCE CLASS TKOF AND LDG NOT
AUTHORIZED EXC FOR ACFT USING APPROVED ALTERNATIVE METHODS OF COMPLIANCE
DUE TO 5G C-BAND INTERFERENCE PLUS SEE AIRWORTHINESS DIRECTIVES
2021-23-12, 2021-23-13 2201190501-2401190501

KDFW 01/114 DFW NAV ILS RWY 13R U/S (Unserviceable) 2301061319-2302212200

KDFW 01/539 DFW TWY A5 HLDG PSN MARKINGS FOR RWY 13R/31L NOT STD 2301192333-2302282359

KDFW 11/926 DFW TWY Q BTN TWY Q4 AND TWY Q6 CLSD 2211291342-2301312300

KDFW 01/588 DFW RWY 18R/36L CLSD 2301231800-2301231900

KDFW 12/159 DFW APRON 1E CORPORATE AVIATION RAMP SPOT 4 CLSD 2212051614-

In the digital age there is literally no excuse for this mess. Software has helped with filtering out what we need but it isn't perfect. Especially when a NOTAM goes off script.

Spork3245 · January 22, 2023

On 1/20/2023 at 10:58 AM, SuperSpreader said:

Now hear me now, there's a concept called backups

From what I understand they had a backup, but the file they needed was corrupted.

b_m_b_m_b_m · January 22, 2023

37 minutes ago, Link200 said:

The were probably on tape drives from 1969.

This issue was annoying I must say. Took me all day to commute home. That said, NOTAMs need completely changed. Maybe this will kick them in the butt to change them.

BTW these are some examples of NOTAMs.

Hide contents