Yes, We Have a Whole Facility UPS

We’ve given a surprising number of tours lately for colocation where one of the questions brought up was whether or not we provide UPS power to the racks. The simple answer is yes: Roller Network maintains and provides facility-wide UPS power. There is no requirement for a colocaiton customer to provide their own UPS. It’s actually our site policy that all switchgear, batteries, and UPS systems must be isolated in the electrical room separate from other equipment.

Apparently the motivation behind this question is that many of our local competitors do not provide UPS power for colocation. Well, we do, and we see it as part of our job in providing colocation services. If you have to maintain your own UPS and batteries, why colocate in the first place?

Primary DNS and Zone Transfers

We’ve received a couple questions about zone transfer (AXFR) support for our Primary DNS service. Currently we do not have AXFR enabled on the DNS servers that answer requests for Primary DNS zones.

The AXFR limitation is not intentional; unfortunately it’s supported by the DNS server. Primary DNS is served using PowerDNS authoritative server version 2.9.22. It does not support per-zone AXFR ACLs like BIND9 does.

The only option we have would be to allow AXFR to anyone, but for security reasons we don’t want to do that. We can’t globally allow our Secondary DNS to AXFR because that creates a backdoor where anyone with an account could use it as an intermediate AXFR host. (Security aside, most of our customers don’t want AXFR open to the world or IP addresses they don’t control.)

PowerDNS authoritative 3.0 is currently in RC2 stage. This version does have per-zone AXFR capability and we will support it in the control center as soon as possible. It also has other features we’d like to add like DNSSEC and long TXT records.

We’re working on changes to the control center required to support these features but ultimately it requires the servers themselves to be upgraded to either the release candidate or the 3.0 release to support them.

Where we are with DNSSEC

From time to time we get questions on DNSSEC support. There are many parts to DNSSEC, but here’s we we stand as of this post:

Our Secondary DNS service (which is based on BIND9) has supported DNSSEC for several years and we have received confirmed reports from some of our customers that use the secondary service that it does work. The Primary DNS service does not support it at this time since it’s based on a version of PowerDNS that lacks DNSSEC support. However, the next release version of PowerDNS will have it, at which point we can work on integrating it into our control center.

On the network side we do not employ any type of mechanisims that try to be “smart” with manipulating DNS traffic incorrectly. Further to that, both UDP and TCP are open for DNS traffic. Contrary to popular belief, DNS queries can use TCP for queries other than AXFR if the UDP query failed, so we allow both.

Generator Test Run: UPS View

Just over a month ago we posted a video of our generator and transfer switch running through the self test process. These two videos show what happens in the UPS room when the transfer switch does its thing. But what happens with the UPS? The first video shows the corresponding UPS actions when the transfer switch moves from utility to generator.

The UPS will go to battery briefly as the transfer (although we don’t see any effect in the room lights) is enough to be momentarily out of tolerance. This is good: the transfer switch will reconnect under minimal load, extending its life and reducing the risk of a contact becoming fused. The generator test runs for an hour, so after that hour is up the transfer switch moves back to utility power from generator. This second video shows the actions of the UPS during the second half of this process.

As you can see, we perform a true test of our emergency power systems with confidence in its operation since a utility outage is identical to our test procedure. During a real utility outage the only major difference is that the UPS would run on battery from 10 to 20 seconds longer while the generator goes through its self start procedures before a transfer.

Generator Test Run Video

I made a quick improvised video of our automatic generator test cycle using my cell phone just for fun this morning for a bit of an inside look at Roller Network and the general sequence of how our generator self test operates. The common practice is to test monthly without load, but simply idling a generator doesn’t indicate it will work during a real outage. Diesels also don’t like to be idled and shut off; they like to work loaded. Unless you take the system on load you aren’t actually testing the whole transfer path.

The most common fault for datacenter transfer switches is failing to transfer. By taking ours on load for tests, we can prove this critical junction between utility and backup power is fully functional.

I recorded this on my EVO which unfortunately doesn’t adjust focus once it starts recording, so everything but the display screens are blurry.