Cisco’s wireless has been on the fritz and the cause might be something out of this world — cosmic radiation, to be exact. After weeks of poor service despite fine fiber specs, Redditor “50cal_vs_squirtgun” got frustrated and went on /r/networking to figure out the bug — and it turns out, Cisco wasn’t full of it.
“Possible trigger is cosmic radiation causing SEU soft errors,” the eventual report read.
While it might sound far-fetched, cosmic radiation is a known scientific cause of electrical surges and can cause what are known as “soft errors” in signal systems. While the cosmic radiation doesn’t break the system, it can cause that sort of network outage.
Scientists believe the cosmic radiation might get bounced off nearby star systems. Studies by IBM in the 1990s suggest that computers typically experience about one cosmic-ray-induced error per 256 megabytes of RAM per month, however, scientists have speculated that outages could increase as chip sizes get smaller. Where bit errors such as those caused by cosmic radiation can sometimes impact two units of data, putting a buffer between cells can prevent the error from spreading.
“It’s gotten a bad rep as it’s not well explained and it’s not the be-all and end-all of outages,” one engineer commented on the Reddit thread. “Cosmic radiation does not home in on a specific part of your box…. It would also hit the control plane and other parts. ECC memory tends to make this a non-issue.”
While the science of the phenomena is sound, technology today more or less prevents the occurrence. It’s unclear why Cisco hasn’t managed to improve their system since the 1990s, or at least the early 2000s.
In a response to a story NetworkWorld wrote on the matter, Cisco released the following statement:
While we can’t speak to this particular case, Cisco has conducted extensive research, dating back to 2001, on the effects cosmic radiation can have on our service provider networking hardware, system architectures and software designs. Despite being rare, as electronics operate at faster speeds and the density of silicon chips increases, it becomes more likely that a stray bit of energy could cause problems that affect the performance of a router or switch.
Luckily, as 50cal_vs_squirtgun found out, the fix for getting the network back up is pretty simple: reload the line card, and you’re star-radiation free.