Charter (Spectrum) Outage Oct. 24-26 Report

Beginning on October 24th at 02:36:56 PDT during a scheduled maintenance window, our circuit to Charter (Spectrum) AS20115 entered loss of service state and was not restored until 57 hours later. Roller Network is disappointed with the delayed response from Charter (Spectrum) to an issue that was created by their own maintenance activities, and the failure of the maintenance group to ensure circuits they removed from service are restored following such activities.

  • At 10:02 we called to notify Charter (Spectrum) that our circuit never recovered following maintenance, in which Charter (Spectrum) intentionally placed the circuit into a loss of service state. We were informed that Charter (Spectrum) will not individually troubleshoot our issue because there was a possible related outage and referenced ticket 50233491.
  • At 14:45 Oct. 24 we again called asking for an ETA on handling out outage. No ETA was given, however we insisted that a ticket was opened and linked to ticket 50233491 (ticket 50234369). We were assured that someone would follow up with us (this did not happen).
  • The following day at 09:26 on Oct. 25 we called to inquire 1) why our circuit was still down and 2) why nobody had followed up with us. We were informed that no followup was made because their notes said the circuit was restored the previous night around 9pm. However, it was not actually restored, and we suggested that ideally someone should have contacted us since we had an open ticket and asked us if it was indeed restored.
  • At 12:16 Oct. 25 we called again to inquire on the status. We were informed that according to the notes nobody had looked at it yet since our last call at 09:26. No information as to why not. At this point the circuit has been down for 33 hours.
  • At 12:48 Oct. 25, a full 34 hours after first loss of service, we finally received a callback asking us to verify site power (which of course we have power) before they can send a tech out.
  • At 13:34 Oct. 25 we received a call from a tech indicating they were en route.
  • At around 15:10 Oct. 25 the tech came to the conclusion that the reason for our ongoing outage was that our circuit was migrated to a new core router, however any and all related configuration was discarded with the migration, specifically all of our BGP configuration for both IPv4 and IPv6, which without BGP the circuit is useless.
  • At 15:21 Oct. 25 we placed BGP neighbors into “shutdown” state because if Charter (Spectrum) maintenance or whoever is responsible for such work deleted our configurations, they would have to recreate it and we would require an audit on their new configuration before we can restore BGP in a controlled manner since the circuit can no longer be trusted.
  • At 21:20 Oct. 25 we stopped actively requesting updates while waiting for Charter (Spectrum) to pass the request to whatever group handled new configurations since Charter (Spectrum) maintenance failed to include migration of existing configurations in their process. However, a decision was made overnight by Charter (Spectrum) to un-migrate our circuit back to its original router and original configuration rather than attempt to migrate our configurations to the new router that maintenance performed to cause the loss of service condition.
  • At 11:51 on Oct. 25 we were finally able to obtain a confirmation in writing from Charter (Spectrum) that our circuit was un-migrated and the original configurations that we had last audited with Charter (Spectrum) on August 3rd, and we returned our BGP neighbors to active state. The total outage duration was 2 days, 9 hours (57 hours) from first loss of service to final confirmation that the circuit was restored to its pre-migration condition.

Service was ultimately restored after 57 hours, however internally Charter (Spectrum) does not recognize this since it overlapped two maintenance windows. Since the circuit was physically restored to a location that it was intentionally moved away from, we fully expect Charter (Spectrum) to make a second attempt at a maintenance window for another migration. Whether or not Charter (Spectrum) will be able to perform this task correctly remains to be seen.

Roller Network disagrees with Charter (Spectrum)’s position that “maintenance” is not responsible for failing to return a circuit to service, and we further assert that whether or not an outage is planned – in this case clearly poorly planned – performing maintenance is still an outage. The sole difference is contractual as to what refunds may be owed or whether or not such could be considered as default of contract. Our circuit went into loss of service state directly due to “maintenance” and was not returned to service, thus “maintenance” is the root cause. From a customer service perspective the ethical course of action would be to cancel any future maintenance and revert all changes performed for failing to complete such within its designated window, rare or not (it was argued that doing so is unnecessary because maintenance failing to successfully complete a task is a “rare” occurrence). Roller Network does not believe it is a customer’s responsibility to make sure “maintenance” performs their job(s) correctly.

Editorial Note: This incident highlights why working with a small business like Roller Network is better than a large company. At no time did our account manager (who was CC’d on all correspondence) offer to step in to help or escalate our case, nor did they follow up to see if our issue was being handled properly. Charter (Spectrum)’s maintenance group, the group one would expect to know exactly what they did to break our circuit, disregarded our issue as a problem for another group since it ceases to be their problem past 6AM even if they fail to restore it working condition by that time. At Roller Network, we do not pass blame between departments, and we always strive make sure our customer’s are in working order – it’s literally our job. Our business with Charter (Spectrum) was treated as unimportant and ultimately irrelevant to them. Charter (Spectrum) is only interested in securing new business for short term gains, disregarding the long term interests of their customers. And that’s the biggest point we can make in our favor: as a small business, when you work with Roller Network you are important to us as an individual on an ongoing, long term basis.

Recent GDPR Stuff

Since everyone and their dog have been posting GPDR updates, we should probably say something about it too.

In a nutshell, Roller Network has never collected or used customer’s personal information. We do not require any personal information to set up an account beyond an email address, and we have never monetized any information. We’ve never had any advertising hooks in our systems whatsoever. We do not have any third party affiliates and we do not engage in data sharing. Information required in the account control center to use a specific service is limited to the scope of that service, and anyone can readily add or delete information as they see fit using it. Realistically, we’re a small company and don’t care about “big data” analytics.

On our colocation side of things, because we don’t offer “cloud” hosting services, our systems do not contain customer data. That’s one major benefit of colocation over cloud: your data and your hardware is yours, it’s not subject to the whim of a larger companies’ policies which – and be honest – can’t to be in your favor because they need to track and monetize your usage very closely.

We also don’t have a default contact preference when signing up for an account: an account can’t be created without choosing one of the three contact options. This means we can be 100% sure that everyone’s contact preference was made intentionally. There’s no check or uncheck the box with confusing wording kind of trickery here that other companies engage in so they can sell your email address with third parties.

Ironically we do occasionally get complaints about having pay for services or why our free accounts are slowly going away. This is why: because we don’t have any other money incoming except customers paying for services. For anyone who does want their personal data shared and monetized to get “free” services, Roller Network is not the place for you, and we’re not planning on changing that.

What we have done is enabled some cookie warnings since it’s harmless, and annoying at worst. We’re also no longer using Google Analytics on our main website and removed the Facebook integration from the Newspipe. We will continue to use Twitter since it is actually useful.

Mail Mirroring as “Email Insurance”

On a semi-regular basis we receive a call or email for help because something has happened to someone’s email: messages were accidentally deleted, their mail server had a config change and rejected everything or accepted and silently discarded messages. Although we do maintain disaster recovery backups, we charge for staff time hourly to try and find and restore a few files without any guarantees to how far back we can look, and that’s only for IMAP; with POP3 the client can remove messages as they are received which never make it into a backup window. Then there’s the SMTP queue: the queue is constantly changing, but since we’re not secretly storing copies of messages just in case, there’s almost no chance to recover anything. In the end, the messages are gone and there’s no simple way to recover them, if at all.

That’s where the Mail Mirror feature comes in. included with every account. A mail mirror uses hosted mail boxes to store copies of messages that pass through our system. Mail Mirror allows you to define addresses or domains to “mirror” to a hosted mail box by storing a copy for backup or emergency access purposes. It uses the independent storage of a normal hosted mail box, which is not affected by the constantly changing mail queue. Once a message goes into a mirror it remains there until it expires based on how long you configure it to keep messages or is manually deleted by logging into the mirror box. This way, a mirror is self-maintaining and won’t keep growing in size. Mail Mirror is available to all accounts and only counts as hosted mail box storage, but for it to work it needs to be enabled before there’s a problem, not after.

Mail mirroring works with all types of mail configurations. You may never need to access your mail mirror, but just like insurance, it’s there just in case.

We’ve also posted a topic to our forums for any questions or discussion on this feature: Mail Mirror – A Helpful Safety Feature

Hurricane Electric POP (2018 Update)

We’ve been talking about the Hurricane Electric POP for a long time now… too long really, but often some of the most difficult things to achieve are the most rewarding.

In summary, a lot of time was spent trying to make Zayo work because at the time, only Zayo could provide an east facing wavelength (cost effectively anyway) to give the HE POP east-west diversity. While the status quo locally has been to backhaul things out of California, doing so increases exposure to the risk of California earthquakes impacting connectivity in an undesirable way. For disaster planning an east facing path is extremely desirable.

The good news is that during all of the time spent working on Zayo, a second option, Verizon, actually improved and is now able to offer an east facing path option to Denver instead of the originally planned Salt Lake City. This is what’s in process now: Zayo is out and Verizon is in. Salt Lake City is out and Denver is in. It’s still going to take a bit more time for Verizon to do the thing they do for long haul, but statistically speaking the number of successful Verizon orders at our facility is significantly higher than successful Zayo orders, so we have a higher confidence level that this is the light at the end of the tunnel.

Hurricane Electric will bring 1Gbps and 10Gbps access ports and PTP transport to their other POPs at prices never seen before in Reno, plus peering at TahoeIX with their famously open peering policy that has made Hurricane Electric one of the top peered networks in the world. Roller Network will be the first neutral colocation facility in Reno a to have a carrier POP* – a real one with peering – not “backhauled bandwidth” or transport to a “city with a router” (for example, all AT&T here goes through a router in Sacramento). We here at Roller Network are excited to be the catalyst for this step away from the status quo.

*Yes, we know there are other bigger fish nearby, but Rollernet is actually in Reno, NV in Washoe County. The others are not, so when we say “Reno” we truly mean within Reno city limits, not somewhere nearby that will never be in Reno. We’re technically correct, the best kind of correct.

Q&A for Mining Colocations

We’ve been receiving a lot of requests for colocating mining hardware lately, so here’s some of the common questions we’ve seen to help speed things up when you contact us.

Do you allow mining in your colo?

Yes, we allow mining hardware in our facility.

Mining doesn’t really need internet, just a 1Mbps connection.

While we understand that mining doesn’t need much internet access, physical ports can’t be shared and require us to support it the same as any other customer, so there’s no cost discount for bandwidth use (or lack of). On the contrary, customers normally expect large amounts of bandwidth for little cost, so we’ve worked hard to make sure bandwidth is only a minor component in the price. As such, there’s no discount by only using a small amount of bandwidth. We also don’t have any NAT in our autonomous system, so each customer uses an IPv4 /30 subnet at minimum, and it some would argue that IPv4 addresses are more valuable than bandwidth.

I only need this for X months.

We only offer term agreements for colocations between 1-year and 5-years.

I only have a small amount of equipment, why do I need full racks?

Only full racks are allowed to colocate pure computing loads. Half racks and shared space are not designed for “density” computing loads. Even if your full rack is mostly empty, it’s the heat footprint that counts.

Furthermore, all mining colocation power will be limited to 5kW per full rack for term lengths less than 3-years. This means if you need more than 5kW you’ll need to spread the load across multiple full racks. We unfortunately can’t allow short term customers to exceed 5kW per full rack.

I only need utility power, can I get a discount?

No, there’s no option for “utility only” power. Our prices include premiums for cooling and backup power (UPS and generator), plus maintenance and testing of those systems.

Why can’t you offer special rates for mining hardware?

Roller Network does not specialize in colocation for mining hardware, we are a general purpose colo facility. We understand there are places offering fixed pricing per physical miner, but we’re not built out to be that kind of place.

How much will it cost to colocate X machines? I don’t know the power.

Our rates aren’t per machine, so you have to know how many kilowatts you need.

Do you charge per kilowatt hour?

No, we don’t offer kWh metering on power at this time.