Category: Status

Categories
Changes Status

System Updates 2020 Edition

Over the next several weeks are are going to try to get all of our systems updated to Debian 9 (oldstable) or ideally Debian 10 (stable) wherever possible. Many systems are still running Debian 8 LTS for which security support ends in June.

We will do our best to minimize disruptions. Any work that’s unavoidably disruptive will be performed during our weekly maintenance window every Saturday between 10:00 and 22:00. An example of a system that’s disruptive would be the account control center, webmail, or IMAP/POP3 hosted mail. To completely update a system there are required reboots, and in some cases (most notably IMAP/POP3 hosted mail), forced switches between active/standby pairs to verify functionality. Updates on our in-office system will cause interruptions to our phone and hotline number (which run on Asterisk) while we perform its upgrades. Email us if you don’t hear the voice prompts answer since someone will be watching for emails when we know voice is being worked on. Work to non-interactive systems like ns1/ns2 may be performed at any time since our supported configuration is that customers configure both on their domain, not just one. We may also decide to migrate older hardware based systems to a virtual machine on a case-by-case basis.

If you have any questions please contact support.

Categories
Changes Status

IP Address Change for Mail Forwarding Servers

The IP addresses of our mail forwarding servers will be changing:

mxfwd1.rollernet.us

  • Old IPv4 Address: 208.79.241.114
  • Old IPv6 Address: 2607:fe70:0:16::a
  • NEW IPv4 Address: 208.79.240.12
  • NEW IPv6 Address: 2607:fe70:0:3::f
  • NEW Name: mxfwd-a.rollernet.us

mxfwd2.rollernet.us

  • Old IPv4 Address: 208.79.241.115
  • Old IPv6 Address: 2607:fe70:0:16::b
  • NEW IPv4 Address: 208.79.241.12
  • NEW IPv6 Address: 2607:fe70:0:4::f
  • NEW Name: mxfwd-b.rollernet.us

If you have created whitelists or used these servers in SPF records (we will update include:m._spf.rollernet.us accordingly) please make sure to add the new addresses alongside the old addresses while this transition is in progress. Once completed, the old IP addresses will no longer be used for any mail-related functions.

CNAME records will be added pointing the legacy names to the new names, so it will be safe to continue referencing the old names.

If you are not using the Mail Forwarding functions in the account control center you will not be affected by this change. Log in to your account and see https://acc.rollernet.us/mail/mapping.php to check if you have mail forwarding configured.

The physical servers are being retired and their mail-related functions replaced with virtual machines. We’ll be repurposing the subnet for timing services since the forwarding servers were also used for NTP (ntp.rollernet.us) and installing Rubidium-based timing systems. This will ensure that functions that are more DNS friendly SMTP functions will transition smoothly, and NTP configurations which are normally configured by IP or only resolved in DNS once will continue with no impact.

UPDATE: All changes were completed successfully.

Categories
Status

January 31, 2020: Degraded Service for Wireless Customers

At approximately 14:00 US/Pacific time and lasting for approximately 45 minutes, some wireless customers were affected by a Cisco IOS bug that was causing some prefixes not to be inserted into TCAM on the preferred (shortest path) core router connected to the serving POP, resulting in null routing for affected prefixes.

Once we were aware of the issue the affected core router was removed from service for analysis. Thanks to the fully redundant nature of our core and backhaul design, this allowed all traffic to self-reroute over alternate paths.

The scope of this issue was limited to wireless customers connecting to a specific POP in Sparks, NV. No other services were affected.

Categories
Status

Annual UPS Preventative Maintenance Visit (2019)

Our annual UPS preventative maintenance visit from Eaton for all facility UPS systems will be on Saturday, October 26th beginning at 09:00 Pacific time and lasting until it’s done (usually mid-afternoon).

As with all routine preventative maintenance visits, we do not anticipate any impact to the facility. If necessary, critical loads will be transferred to generator.

UPDATE: Annual maintenance completed on all units with no issues. (Oct. 26 @ 13:53 PDT)

Categories
Changes Status

Rejecting BGP RPKI “Invalid” Prefixes

Roller Network AS11170 will be updating our routing policy to reject any IPv4 or IPv6 prefix with a BGP RPKI validation result of “invalid” on both the peering and transit borders of our network. We’ve been running RPKI validation internally for a while with the “bgp bestpath prefix-validate allow-invalid” setting configured. This routing policy change will simply remove this line from our BGP address family configurations.