Quad9 DNS

Have you tried Quad9 DNS yet? It’s run by PCH Global, they’ve got a local node right here in Reno, and they’re peered on TahoeIX.

  • Set your DNS server to to use Quad9 DNS
  • Quad9 has IPv6 support at 2620:fe::fe
  • Quad9 supports DNS over TLS on port 853 (the standard) using an auth name of dns.quad9.net
  • Quad9 also supports DNS over HTTPS using the query https://dns.quad9.net/dns-query

For more information see https://www.quad9.net/faq/

Charter Second Maintenance Attempt

Charter (Spectrum) has notified us they’re going to try maintenance again Thu 6 Dec 2018 12:00 AM – Fri 7 Dec 2018 6:00 AM with a claimed 240 minutes loss of service during this timeframe.

Since the previous attempt went extremely poorly and resulted in 57 hours loss of service, we can only hope their maintenance group is prepared for this second attempt. We will provide updates as necessary.

UPDATE 12/6/2018: Charter (Spectrum) tried to migrate our circuit again, and now IPv6 is broken due to BGP failing to establish.

UPDATE: Charter (Spectrum) IPv6 BGP was finally restored on December 10 at 22:01.

Charter (Spectrum) Outage Oct. 24-26 Report

Beginning on October 24th at 02:36:56 PDT during a scheduled maintenance window, our circuit to Charter (Spectrum) AS20115 entered loss of service state and was not restored until 57 hours later. Roller Network is disappointed with the delayed response from Charter (Spectrum) to an issue that was created by their own maintenance activities, and the failure of the maintenance group to ensure circuits they removed from service are restored following such activities.

  • At 10:02 we called to notify Charter (Spectrum) that our circuit never recovered following maintenance, in which Charter (Spectrum) intentionally placed the circuit into a loss of service state. We were informed that Charter (Spectrum) will not individually troubleshoot our issue because there was a possible related outage and referenced ticket 50233491.
  • At 14:45 Oct. 24 we again called asking for an ETA on handling out outage. No ETA was given, however we insisted that a ticket was opened and linked to ticket 50233491 (ticket 50234369). We were assured that someone would follow up with us (this did not happen).
  • The following day at 09:26 on Oct. 25 we called to inquire 1) why our circuit was still down and 2) why nobody had followed up with us. We were informed that no followup was made because their notes said the circuit was restored the previous night around 9pm. However, it was not actually restored, and we suggested that ideally someone should have contacted us since we had an open ticket and asked us if it was indeed restored.
  • At 12:16 Oct. 25 we called again to inquire on the status. We were informed that according to the notes nobody had looked at it yet since our last call at 09:26. No information as to why not. At this point the circuit has been down for 33 hours.
  • At 12:48 Oct. 25, a full 34 hours after first loss of service, we finally received a callback asking us to verify site power (which of course we have power) before they can send a tech out.
  • At 13:34 Oct. 25 we received a call from a tech indicating they were en route.
  • At around 15:10 Oct. 25 the tech came to the conclusion that the reason for our ongoing outage was that our circuit was migrated to a new core router, however any and all related configuration was discarded with the migration, specifically all of our BGP configuration for both IPv4 and IPv6, which without BGP the circuit is useless.
  • At 15:21 Oct. 25 we placed BGP neighbors into “shutdown” state because if Charter (Spectrum) maintenance or whoever is responsible for such work deleted our configurations, they would have to recreate it and we would require an audit on their new configuration before we can restore BGP in a controlled manner since the circuit can no longer be trusted.
  • At 21:20 Oct. 25 we stopped actively requesting updates while waiting for Charter (Spectrum) to pass the request to whatever group handled new configurations since Charter (Spectrum) maintenance failed to include migration of existing configurations in their process. However, a decision was made overnight by Charter (Spectrum) to un-migrate our circuit back to its original router and original configuration rather than attempt to migrate our configurations to the new router that maintenance performed to cause the loss of service condition.
  • At 11:51 on Oct. 25 we were finally able to obtain a confirmation in writing from Charter (Spectrum) that our circuit was un-migrated and the original configurations that we had last audited with Charter (Spectrum) on August 3rd, and we returned our BGP neighbors to active state. The total outage duration was 2 days, 9 hours (57 hours) from first loss of service to final confirmation that the circuit was restored to its pre-migration condition.

Service was ultimately restored after 57 hours, however internally Charter (Spectrum) does not recognize this since it overlapped two maintenance windows. Since the circuit was physically restored to a location that it was intentionally moved away from, we fully expect Charter (Spectrum) to make a second attempt at a maintenance window for another migration. Whether or not Charter (Spectrum) will be able to perform this task correctly remains to be seen.

Roller Network disagrees with Charter (Spectrum)’s position that “maintenance” is not responsible for failing to return a circuit to service, and we further assert that whether or not an outage is planned – in this case clearly poorly planned – performing maintenance is still an outage. The sole difference is contractual as to what refunds may be owed or whether or not such could be considered as default of contract. Our circuit went into loss of service state directly due to “maintenance” and was not returned to service, thus “maintenance” is the root cause. From a customer service perspective the ethical course of action would be to cancel any future maintenance and revert all changes performed for failing to complete such within its designated window, rare or not (it was argued that doing so is unnecessary because maintenance failing to successfully complete a task is a “rare” occurrence). Roller Network does not believe it is a customer’s responsibility to make sure “maintenance” performs their job(s) correctly.

Editorial Note: This incident highlights why working with a small business like Roller Network is better than a large company. At no time did our account manager (who was CC’d on all correspondence) offer to step in to help or escalate our case, nor did they follow up to see if our issue was being handled properly. Charter (Spectrum)’s maintenance group, the group one would expect to know exactly what they did to break our circuit, disregarded our issue as a problem for another group since it ceases to be their problem past 6AM even if they fail to restore it working condition by that time. At Roller Network, we do not pass blame between departments, and we always strive make sure our customer’s are in working order – it’s literally our job. Our business with Charter (Spectrum) was treated as unimportant and ultimately irrelevant to them. Charter (Spectrum) is only interested in securing new business for short term gains, disregarding the long term interests of their customers. And that’s the biggest point we can make in our favor: as a small business, when you work with Roller Network you are important to us as an individual on an ongoing, long term basis.

Recent GDPR Stuff

Since everyone and their dog have been posting GPDR updates, we should probably say something about it too.

In a nutshell, Roller Network has never collected or used customer’s personal information. We do not require any personal information to set up an account beyond an email address, and we have never monetized any information. We’ve never had any advertising hooks in our systems whatsoever. We do not have any third party affiliates and we do not engage in data sharing. Information required in the account control center to use a specific service is limited to the scope of that service, and anyone can readily add or delete information as they see fit using it. Realistically, we’re a small company and don’t care about “big data” analytics.

On our colocation side of things, because we don’t offer “cloud” hosting services, our systems do not contain customer data. That’s one major benefit of colocation over cloud: your data and your hardware is yours, it’s not subject to the whim of a larger companies’ policies which – and be honest – can’t to be in your favor because they need to track and monetize your usage very closely.

We also don’t have a default contact preference when signing up for an account: an account can’t be created without choosing one of the three contact options. This means we can be 100% sure that everyone’s contact preference was made intentionally. There’s no check or uncheck the box with confusing wording kind of trickery here that other companies engage in so they can sell your email address with third parties.

Ironically we do occasionally get complaints about having pay for services or why our free accounts are slowly going away. This is why: because we don’t have any other money incoming except customers paying for services. For anyone who does want their personal data shared and monetized to get “free” services, Roller Network is not the place for you, and we’re not planning on changing that.

What we have done is enabled some cookie warnings since it’s harmless, and annoying at worst. We’re also no longer using Google Analytics on our main website and removed the Facebook integration from the Newspipe. We will continue to use Twitter since it is actually useful.

Mail Mirroring as “Email Insurance”

On a semi-regular basis we receive a call or email for help because something has happened to someone’s email: messages were accidentally deleted, their mail server had a config change and rejected everything or accepted and silently discarded messages. Although we do maintain disaster recovery backups, we charge for staff time hourly to try and find and restore a few files without any guarantees to how far back we can look, and that’s only for IMAP; with POP3 the client can remove messages as they are received which never make it into a backup window. Then there’s the SMTP queue: the queue is constantly changing, but since we’re not secretly storing copies of messages just in case, there’s almost no chance to recover anything. In the end, the messages are gone and there’s no simple way to recover them, if at all.

That’s where the Mail Mirror feature comes in. included with every account. A mail mirror uses hosted mail boxes to store copies of messages that pass through our system. Mail Mirror allows you to define addresses or domains to “mirror” to a hosted mail box by storing a copy for backup or emergency access purposes. It uses the independent storage of a normal hosted mail box, which is not affected by the constantly changing mail queue. Once a message goes into a mirror it remains there until it expires based on how long you configure it to keep messages or is manually deleted by logging into the mirror box. This way, a mirror is self-maintaining and won’t keep growing in size. Mail Mirror is available to all accounts and only counts as hosted mail box storage, but for it to work it needs to be enabled before there’s a problem, not after.

Mail mirroring works with all types of mail configurations. You may never need to access your mail mirror, but just like insurance, it’s there just in case.

We’ve also posted a topic to our forums for any questions or discussion on this feature: Mail Mirror – A Helpful Safety Feature