This is a short story about automation, brittle systems, and the quiet ways complex technology fails.
“Collision Warning! Collision Warning!” The alarms blared in the control center for Perihelion Dynamics, a startup that had deployed a constellation of low Earth orbit satellites. One of a growing number of such constellations that have formed in recent years. “Flight Director! 20158 is going the wrong way” the GNC Lead yelled, “it’s going straight towards an Amazon satellite. 20158 triggered an avoidance burn, but solution is inverted.”
When the typically quiet Guidance, Navigation, and Control lead yelled, everyone paid attention. A reserved woman with dual doctorates, in physics and math; a calm and cool professional, unless something was very, very wrong.
“Give me 20158 on the main display” the Flight Director barked. A theatre-sized screen switched from showing all 10,000 satellites in the Perihelion Dynamics mega-constellation, to a close-up view of a satellite. A satellite moving perilously close to a collision.
“GNC, any idea what’s going on here?”
“FD, no. Avoidance burn auto-triggered at 20km distance. Burn completed, but in opposite orientation. It’s like the delta-v is inverted. Burns are exactly opposite of what they should be. Continuing to close on Amazon Leo 58739, currently at 2.2km. Collision projected in 55 seconds.”
“This is Flight Director, is the Space Traffic Liaison on the net?”
“FD, STL here. Reaching out to the Amazon Leo C&C. Requesting they perform emergency burn.”
“GNC, switch to manual and execute a manual burn.”
“FD, activating emergency override and switching to manual navigation. Starting 5 second burn, full emergency thrust. Burn started.” She spins a ring on her finger as she watches the screen. Waiting to see the thrusters fire. “No. No. It started the wrong thruster. Closing speed increasing. Collision imminent. Distance 0.4km.”
“STL, any update from the Leo C&C?”
“FD, Leo C&C has issued burn command. Leo 58739 should begin moving away.”
“GNC, confirm if we are clearing.”
“FD, Leo 58739 course change detected. Collision probably dropping. Collision avoidance system reset auto-triggered. Standby. No. No. Collision avoidance burn auto-triggered due to new course. Solution is again inverted. Collision probably climbing. Distance 100 meters and closing.” She waited and watched, fidgeting with the ring on her index finger. The seconds dragged by. “Telemetry lost.”
“This is Flight Director, is the Mission Director on the net?”
“FD, MD here.”
“MD, we have an incident. Satellite collision.”
It’s been 20 years since the first mega-constellation was launched in low Earth orbit. Constellations with 5,000 or more satellites have become common. Three are based in the United States, two in China, and two in Europe. Well over 100,000 of these LEO satellites circle the Earth, providing communications and compute services.
With orbit becoming so congested, these satellites all used sophisticated collision avoidance systems, with each constellation rarely going more than a few minutes without at least one thruster burn to avoid hitting another satellite or debris. These are generally handled automatically and without any human intervention.
Experiments in large-scale compute in orbit have resulted in these constellations growing both in terms of number of satellites, and in the size of the satellites. High-end compute satellites are now common in polar orbits, with truly massive solar arrays – sometimes exceeding 25 square kilometres. These massive satellites have made managing traffic in orbit far more complex.
Resistance to these deployments, for everything from destroying the ability to use ground-based telescopes, to the millions of pounds of metals being vaporised in the upper atmosphere, is growing louder and louder. Rockets found with holes from sniper rifles have long since stopped being a surprise.
“FD, this is GNC. Radar is showing approximately 900 new pieces of debris from the location of 20158. Collision avoidance updates are propagating. We are seeing burns triggered on multiple satellites.”
“GNC, acknowledged. Please confirm that burn solutions are correct. I don’t want to see a repeat of 20158.”
“FD, please standby. 20159, solution is inverted. 20160, solution is inverted. 20161, solution is inverted. 19476, solution is inverted. 19584, solution is inverted. The satellites are moving towards the debris field. I don’t understand what’s happening, but all burns are inverted.”
“GNC, say again. Satellites in the area are moving closer to the tracked debris?”
“FD, confirmed. Automated and manual navigation commands are resulting in inverted burn solutions.”
“MD, this is FD, did you copy that?”
“FD, affirm. Declaring Sev1 incident. Initiate emergency procedures. I’m leaving the net to contact the Executive Duty Officer.”
“MD, copy. All on the net: This is Flight Director. We have a Severity 1 incident. Emergency procedures are now in effect. Secure all external doors. External communications are restricted. Ground Systems Lead, Fleet Software Lead, Space Traffic Liaison, Licensing & Compliance Lead, Security Operations Lead are all to join and remain on net. GNC, please provide current status.”
“FD, current status. We have 5 satellites that are moving to a debris field. We have 1 satellite that is moving closer to a China Telecom satellite. We have 1 that is moving closer to tracked debris. All navigation commands result in inverted burn solutions. We do not have effective navigation control over the constellation. The situation is actively devolving. Currently showing 7 satellites with a greater than 90% chance of collision. Satellite 20159 is showing collision imminent.”
“This is Flight Director, is the Fleet Software Lead on the net?”
“FD, this is FSL.”
“FSL, I need to know what’s going on with the navigation software. Now.”
“FD, Fleet Software team is reviewing logs and telemetry now, will advise when we have further information.”
“FSL, can you give me a workaround to bypass whatever is inverting the burn solutions?” He knew that there was no real chance that the answer to this would be anything other than being told to wait, but all he could do at this point was ask. In this moment, there was nothing at all he could do.
“FD, negative at this time. We are not yet aware of why these commands aren’t being processed as intended. Will notify when we know more.”
The Flight Director walked over to a window in the back of the control center, and looked up at the evening sky. Seemingly peaceful, seemingly calm. So many times he had looked up at the sight, and dreamed of what it would be like to be there. Now, that thought sent a shiver down his spine. It looked calm to his eyes, but he knew that things were becoming anything but calm.
Deep in the navigation code that Perihelion Dynamics had built, a tiny change had been slipped in. A change easily overlooked. A change so vital, so important, that it would change everything.
- let bp = calculate_burn_params(&sat_state, desired_delta_v, thrust_limit);
+ let bp = calculate_burn_params(&sat_state, -desired_delta_v, thrust_limit);
A single character was added, a minus, reversing the intended thrust. When this new code was called, it would trigger a burn going in the opposite direction. This small change doomed the satellites.
Discovering this change would take far more time than they actually had. Because of how the change was introduced, it will take weeks to discover this change. By then, it will be far too late.
The Fleet Software Lead and her team, a group of 15 people, were staring at logs and telemetry that made no sense. The navigation commands were clear, yet the wrong thrusters were being used. They fed the same data into their simulators and test suites – in each case the proper thrusters were used. They reviewed the code line by line, yet the code they had was correct.
Hour after hour, they searched, tested, searched again, and tested more. Nothing made sense. Everything should be working. Everything was working before. They reviewed every change made in their code, and none would cause this.
Little did they know, the code they saw wasn’t the code that was running on the satellites.
“Attention on the net. This is the Executive Duty Officer. We’re six hours into a Sev1, can anyone tell me anything useful about what’s going on? I’ve been on the phone with NASA, US Space Force, ESA, JAXA, CNSA, three Senators, two MPs, 5 CEOs, and every one of them has done nothing but yell. Please tell me we know what’s happening.”
“EDO, this is the Mission Director. We’ve lost 17 of our satellites, and 9 belonging to others have been lost in the process. Due to the new collisions, several thousand new pieces of debris are being tracked, though we don’t know the exact numbers. One of our polar orbit compute satellites was just lost. That unit had a 9 square kilometre solar array. The debris field is expansive. Collateral damage is steadily increasing. Other satellites aren’t able to move fast enough to avoid the amount of debris being generated. Our satellite navigation software doesn’t support sending manual thruster commands, only directional commands. We are having some luck by sending manual commands that are already reserved, though with the expanding debris clouds, this has limited effectiveness.”
The Mission Director’s words were followed by a muffled thud. The sound of the Executive Duty Officer’s headset flying across his office.
In the weeks before the first satellite malfunctioned, a group of hactivists began targeting developers that worked for Perihelion Dynamics with carefully crafted phishing emails. Emails claiming to be from a recruiter for a competitor that was known to pay substantially more. One developer downloaded an attachment, part of a skill assessment, on his work laptop. They were in.
The attackers carefully explored and documented Perihelion Dynamics’ systems and software. They studied the code. They analysed the development infrastructure. Then they turned their attention to their build servers – the servers that took the code and prepared it for deployment on the satellites. This is where they would strike.
With a small change to the build server, they added an additional step to the process: apply the 1-line change to a code file to reverse the thrust calculation. Once this patch was applied, the code would be built as normal, and automatically deployed to the satellites. The developers would never see the modified version of the code.
A silent attack that would be very difficult to notice. Only when the binary sent to the satellites was examined would the change become visible.
The statement “telemetry lost; debris detected” was a common refrain not only in the Perihelion Dynamics control center, but in the control centres of satellite operators around the world. It started with 2 satellites turning into hundreds of pieces of debris, and then more satellites turned into even more debris. Debris traveling at more than 17,000 miles per hour.
Over the next 24 hours, the situation turned into a catastrophic nightmare. Dozens of satellites lost turned into hundreds, then hundreds turned into thousands. A thick layer of debris was building up around the Earth; from 500 to 700 kilometres above the surface, it was hard for anything to survive.
Private space stations were abandoned – at least two were destroyed. In the chaos, it was hard to be certain of what was still intact. Almost everything was damaged. What started as hundreds of pieces of new debris turned into thousands, then into millions.
The large compute satellites were all but gone. Given their large size and massive solar arrays, it was simply impossible to dodge the ever growing debris field. With thousands of solar panels that covered several square kilometres, they were quickly added to the debris cloud.
The control center was a hive of activity. The teams were exhausted, stumped, spent, but still fighting. The Guidance, Navigation, and Control lead had finally developed a feel for sending reversed navigation commands to minimise collisions. It didn’t always work, but she was saving what she could.
The radar onboard the satellites was consistently overwhelmed by the amount of debris. The software wasn’t able to effectively track it all. The GNC Lead and her team had to manually sort out thruster burns to find paths that may still be open. The software had given up on finding safe paths among the masses of debris.
The GNC Lead had now been working for 25 hours, only stepping away for a few minutes at a time.
“Attention on the net. This is the Executive Duty Officer. By order of the US Government and the European Commission, we are to manually de-orbit all remaining satellites. It is understood that given the level of debris, we may not be able to fully de-orbit, though the intent is to push satellites and debris to the lowest possible orbit. A similar order has been issued to other operators. All possible satellites that can be removed from orbit, are to be removed.”
Everyone listening sat in a stunned silence. They were giving up. After 24 hours of constant work, after thousands of manual burns to dodge debris, after fighting their own software constantly to save what they could, this was the end. The end for them, and for everyone else.
This was just the start of government action. Flights were grounded due to fears of debris reaching the ground. All space launch licenses were cancelled. Telescopes on the ground were darkened – there was too much debris to perform any useful science. For years, saying “because we can” was justification. In a day, the bill for that recklessness had come due.
After nearly 48 hours, the doors were opened at the Perihelion Dynamics control center. The Flight Director and GNC Lead sat on a bench and watched the night sky. There seemed to be twice as many stars as there used to be. So many twinkling lights. A sky more full than they had ever seen.
“The first debris will start coming down soon,” the Flight Director said.
“Yes. And it won’t stop for 10 years. Once it starts, we’ll see debris burning up every night. It’ll be a constant fireworks show for at least 5 years.” The GNC Lead responded sadly.
“I wonder what’ll happen to the moon base” he asked.
“They’ll be alone for the next 15 years.”
