Categories
Announcements Status

DST Root CA X3 Expired

For more information from Let’s Encrypt visit:
https://letsencrypt.org/docs/dst-root-ca-x3-expiration-september-2021/


Since September 30th we have received reports from two of our mail services customers that they were no longer able to send/receive mail with their Exchange servers due to a certificate expiration error. Our certificates have not expired; a root certificate (DST Root CA X3) has expired. The Let’s Encrypt R3 is signed by DST Root CA X3 (now expired) and ISRG Root X1 (trusted). The latter, ISRG Root X1, is what should be used.

Since June 2017 Roller Network has been using Let’s Encrypt. Our certificates update continuously: the certificates we have are only valid for 90 days, then are automatically replaced with a new one when they reach less than 30 days until expiration. It’s not possible for us to have “old” certificates since the oldest one will only ever be 2 months old before it gets replaced with a new one (and we monitor every Let’s Encrypt deployed service for freshness with alerts if a cert goes under 28 days). This process happens continuously on every system we have that uses SSL/TLS.

All certificates ultimately rely on a chain of trust based on a root store of trusted certificates present in every platform that the chain of validation is based on. All of these also have an expiration date, but a longer one since changing these either requires an OS update (usually in the form of security updates) or for platforms that no longer receive updates, manually installing new root certificates if it doesn’t have ISRG Root X1 installed. Alternatively, some platforms allow manually setting trust for an expired root certificate or need to remove an old root certificate.

The only reports we have received with TLS problems is with Exchange. Unfortunately we don’t have anyone on staff with Exchange experience, so we don’t have a fix to give out. At this point we can only recommend reading what others have done to address issues with cross-signed certificate authorities, although if we find a procedure specific for Exchange we’ll pass it along. For platforms based on OpenSSL 1.0.x this is a known bug which is fixed by updating to OpenSSL 1.1.x.

The reason the old, expired root is still in the chain is for an Android compatibility thing as detailed here: https://letsencrypt.org/2020/12/21/extending-android-compatibility.html

To view certificates in Windows see:
https://docs.microsoft.com/en-us/dotnet/framework/wcf/feature-details/how-to-view-certificates-with-the-mmc-snap-in

On Windows the expired “DST Root CA X3” should be under Trusted Root Certification Authorities and Third Party Root Certificates. Removing it and rebooting may help (or it may not, we don’t have a way to test an Exchange server).

For a complete list of platforms compatible with Let’s Encrypt see:
https://letsencrypt.org/docs/certificate-compatibility/

Categories
Status

Suite 2 Customer Edge Switch 9/9/2021

On September 9th, 2021 the customer edge switch in Suite 2 experience a watchdog reload while switching from active to standby supervisor after a line card in slot 9 experienced a fabric error, and then required manual intervention to boot (execute “boot” at rommon prompt). The slot 9 line card was reseated. Total duration of the event was about 30 minutes.

This did not affect any cross connects or connections to the Suite 1 customer edge switch, nor was our network core, transit, or peerings impacted.

We will be writing up more details to send impacted customers directly in the coming week or so, along with an overview of redundancy options we have available and what can be improved.

Categories
Announcements TahoeIX

DNS over HTTPS and TLS

We’ve recently enabled DNS over HTTPS (DoH) and DNS over TLS on our resolvers for our customers (IPv4 and IPv6).

For DNS over HTTPS (DoH) use:

https://dns.rollernet.us/dns-query

For DNS over TLS use:

tls://dns.rollernet.us

Our DNS servers validate DNSSEC (queries will be answered with SERVFAIL in case of bogus data). If you have trouble resolving DNS that appears to work with sites that are not DNSSEC-aware, check it with the DNSViz tool: https://dnsviz.net

Our DNS servers will only respond to queries from our network (users on Roller Network IP addresses). If you are a peer or downstream customer with your own address space, please contact us to add your IP addresses to our “allow” list. For public service we recommend Quad9. Quad9 is globally anycasted including a local peer in Reno, NV at TahoeIX. For more information visit: https://www.quad9.net

Categories
Announcements

New Site

New website. Same content. Less pictures (for now but we’ll work on that, maybe). We have redirects for all the old URIs we know of, but let us know if something is broken.

Categories
Changes Status

SSL/TLS Changes

We’re going to start turning off TLSv1.0 and TLSv1.1 per best current practices (BCP 195), and start working on updates to add support for TLSv1.3. Our account control center is first. Other services will be changed as we work on configs or other updates for both web and mail services.

As of early 2020, support for TLS 1.0 and TLSv1.1 has been removed in current versions of major browsers. For more information about the depreciation of TLS1.0/1.1 see: https://blog.qualys.com/product-tech/2018/11/19/grade-change-for-tls-1-0-and-tls-1-1-protocols

We’re also changing our ACME client for Let’s Encrypt certificates. We started out using certbot, however certbot is moving to an app store framework (Snap) for future updates and we don’t want to install such things on our servers. So we searched for an alternative ACME client that we liked and settled on dehydrated. For more information on dehydrated visit them on github at: https://github.com/dehydrated-io/dehydrated