Categories
Announcements Status

DST Root CA X3 Expired

For more information from Let’s Encrypt visit:
https://letsencrypt.org/docs/dst-root-ca-x3-expiration-september-2021/


Since September 30th we have received reports from two of our mail services customers that they were no longer able to send/receive mail with their Exchange servers due to a certificate expiration error. Our certificates have not expired; a root certificate (DST Root CA X3) has expired. The Let’s Encrypt R3 is signed by DST Root CA X3 (now expired) and ISRG Root X1 (trusted). The latter, ISRG Root X1, is what should be used.

Since June 2017 Roller Network has been using Let’s Encrypt. Our certificates update continuously: the certificates we have are only valid for 90 days, then are automatically replaced with a new one when they reach less than 30 days until expiration. It’s not possible for us to have “old” certificates since the oldest one will only ever be 2 months old before it gets replaced with a new one (and we monitor every Let’s Encrypt deployed service for freshness with alerts if a cert goes under 28 days). This process happens continuously on every system we have that uses SSL/TLS.

All certificates ultimately rely on a chain of trust based on a root store of trusted certificates present in every platform that the chain of validation is based on. All of these also have an expiration date, but a longer one since changing these either requires an OS update (usually in the form of security updates) or for platforms that no longer receive updates, manually installing new root certificates if it doesn’t have ISRG Root X1 installed. Alternatively, some platforms allow manually setting trust for an expired root certificate or need to remove an old root certificate.

The only reports we have received with TLS problems is with Exchange. Unfortunately we don’t have anyone on staff with Exchange experience, so we don’t have a fix to give out. At this point we can only recommend reading what others have done to address issues with cross-signed certificate authorities, although if we find a procedure specific for Exchange we’ll pass it along. For platforms based on OpenSSL 1.0.x this is a known bug which is fixed by updating to OpenSSL 1.1.x.

The reason the old, expired root is still in the chain is for an Android compatibility thing as detailed here: https://letsencrypt.org/2020/12/21/extending-android-compatibility.html

To view certificates in Windows see:
https://docs.microsoft.com/en-us/dotnet/framework/wcf/feature-details/how-to-view-certificates-with-the-mmc-snap-in

On Windows the expired “DST Root CA X3” should be under Trusted Root Certification Authorities and Third Party Root Certificates. Removing it and rebooting may help (or it may not, we don’t have a way to test an Exchange server).

For a complete list of platforms compatible with Let’s Encrypt see:
https://letsencrypt.org/docs/certificate-compatibility/

Categories
Status

Suite 2 Customer Edge Switch 9/9/2021

On September 9th, 2021 the customer edge switch in Suite 2 experience a watchdog reload while switching from active to standby supervisor after a line card in slot 9 experienced a fabric error, and then required manual intervention to boot (execute “boot” at rommon prompt). The slot 9 line card was reseated. Total duration of the event was about 30 minutes.

This did not affect any cross connects or connections to the Suite 1 customer edge switch, nor was our network core, transit, or peerings impacted.

We will be writing up more details to send impacted customers directly in the coming week or so, along with an overview of redundancy options we have available and what can be improved.

Categories
Changes Status

SSL/TLS Changes

We’re going to start turning off TLSv1.0 and TLSv1.1 per best current practices (BCP 195), and start working on updates to add support for TLSv1.3. Our account control center is first. Other services will be changed as we work on configs or other updates for both web and mail services.

As of early 2020, support for TLS 1.0 and TLSv1.1 has been removed in current versions of major browsers. For more information about the depreciation of TLS1.0/1.1 see: https://blog.qualys.com/product-tech/2018/11/19/grade-change-for-tls-1-0-and-tls-1-1-protocols

We’re also changing our ACME client for Let’s Encrypt certificates. We started out using certbot, however certbot is moving to an app store framework (Snap) for future updates and we don’t want to install such things on our servers. So we searched for an alternative ACME client that we liked and settled on dehydrated. For more information on dehydrated visit them on github at: https://github.com/dehydrated-io/dehydrated

Categories
Announcements Changes Status

SquirrelMail End of Life Notice

Since the very beginning of our hosted mail service, SquirrelMail was there as a webmail option. It’s served thousands of users well, but the time has come where we need to declare SquirrelMail end of life (EOL) and no longer supported. SquirrelMail has not been actively developed for several years, and incompatibility with other upgrades we will be making across our system is very likely.

As an EOL service, we will no longer test changes for compatibility with SquirrelMail. Although the basic email functions of SquirrelMail will continue to function, any broken additional features in SquirrelMail will be removed or disabled.

We understand the desire for a simple webmail interface still exists in our customer base and we will be looking at replacement options. In the meantime, please plan to migrate your webmail needs to RoundCube at: https://webmail.rollernet.us/roundcube

SquirrelMail will be shut down on December 31, 2020.

Categories
Changes Status

System Updates 2020 Edition

Over the next several weeks are are going to try to get all of our systems updated to Debian 9 (oldstable) or ideally Debian 10 (stable) wherever possible. Many systems are still running Debian 8 LTS for which security support ends in June.

We will do our best to minimize disruptions. Any work that’s unavoidably disruptive will be performed during our weekly maintenance window every Saturday between 10:00 and 22:00. An example of a system that’s disruptive would be the account control center, webmail, or IMAP/POP3 hosted mail. To completely update a system there are required reboots, and in some cases (most notably IMAP/POP3 hosted mail), forced switches between active/standby pairs to verify functionality. Updates on our in-office system will cause interruptions to our phone and hotline number (which run on Asterisk) while we perform its upgrades. Email us if you don’t hear the voice prompts answer since someone will be watching for emails when we know voice is being worked on. Work to non-interactive systems like ns1/ns2 may be performed at any time since our supported configuration is that customers configure both on their domain, not just one. We may also decide to migrate older hardware based systems to a virtual machine on a case-by-case basis.

If you have any questions please contact support.