January 31, 2020: Degraded Service for Wireless Customers

At approximately 14:00 US/Pacific time and lasting for approximately 45 minutes, some wireless customers were affected by a Cisco IOS bug that was causing some prefixes not to be inserted into TCAM on the preferred (shortest path) core router connected to the serving POP, resulting in null routing for affected prefixes.

Once we were aware of the issue the affected core router was removed from service for analysis. Thanks to the fully redundant nature of our core and backhaul design, this allowed all traffic to self-reroute over alternate paths.

The scope of this issue was limited to wireless customers connecting to a specific POP in Sparks, NV. No other services were affected.