November, 2009:

Outage Notice Nov. 22, 2009

There was a power outage today at 11:57 UTC-8 caused by this:

DCP_2642

We’re actually about a mile away from it, but as you can see it hit the end of the line, popped the top middle phase, and tripped the whole line we happen to be on. Everything was fine until 14:33 UTC-8 when our new Eaton 9355 installation decided to cut power to the protected load. The alarms it gave were simultaneous “Input AC Over Voltage”, “Utility Fail” followed by “DC Link Over Voltage”. We’re guessing the generator momentarily went outside of tolerance. We’re not clear why that would cause the UPS to shut off (isn’t this the whole point of the UPS?), but it refused to turn back on and kept giving “DC link over voltage”. Because we were offline at this point, we activated the external bypass on the paralleling cabinet to tie the generator directly to the server room distribution and get things back online ASAP. We called the Eaton support hotline and the tech advised us to reset the logic by cycling all the breakers open then closed, however the unit still failed to start citing the “DC link over voltage” alarm. We do have a 24 hour service contract for the unit, so someone is on the way to take a look at it.

[Nov. 23 00:49] Eaton took a look at the unit and was unable to duplicate the problem. The only difference is that it was supplied by utility power and not by generator as we do not wish to experiment without a dummy load and risk an outage. At this time the bypasses have been removed, the generator stopped, and the UPS is back online supplying power to the protected load. We will also have the generator looked at as we suspect the most likely cause of the trouble could be the voltage regulator losing lock or the electronic governor. The units we have are transformerless and the “DC link over voltage” error can be caused by a problem with the neutral reference point going out of bounds.

[Nov. 23 13:00] Our diesel mechanic will be replacing all of the injectors on our generator. One of them had a blown port due to water.

[Nov. 24 10:28] According to the news, the driver of the van dropped something. (Source: KOLO) West Huffaker Lane is 5 lanes wide, four travel and one center turn lane.


View Outage Notice Nov. 22, 2009 in a larger map

Changes to “Client Validity Checks” Filter

We made two changes to the Client Validity Checks filter:

1) All test options now have both reject and defer actions available.

2) A “strict” option was added for the HELO name test. The existing test would include subdomains, while the strict option will only perform an exact match.

Perl API Interface

Keith Wessel has graciously provided a Perl module interface to our control center API. We’ll be posting a download link on the API help page soon, but for now you can get it here: RollernetClient.pm Version 1.0

New Facility Progress #11

DCP_2616

A generator awaiting installation.

DCP_2627

The “rollers”, our unofficial mascots that take residence in the server room on top of a router.

DCP_2638[1] DCP_2637

Detail view of the electrical outlet installation. The smallest circuit we offer is 20A at 208VAC in single phase or three phase, but we can easily provide multiple 20 and 30 amp circuits per rack in single phase or three phase for high density racks. (We do not offer 120VAC circuits.)

DCP_2631

Detail view of the patchrack. By keeping patch panels out of the racks and using the overhead ladder to route them where they need to be we increase server density and eliminate wasted ports. If high port density per rack is required, top-of-rack switching would be used.

DCP_2640

Detailed view of the overhead area. Even though you can’t see this without a ladder, we like to keep everything in order.

DCP_2632

Power cord cable management for the router and switch racks.

DCP_2635

A view of the server room. What was spread out over five racks in the old facility was consolidated into two racks (not counting the two-post racks to the right). The extra space will be offered for colocations and dedicated servers and we will continue installing racks to fill the room. The yellow cables hanging out of the wire management on the right indicate temporary cabling.

Outbound Mail Now IPv6 Enabled

We recently enabled IPv6 in Postfix on our outgoing mail (SMTP AUTH, smarthost) service. This means that if a destination you’re sending mail to is accessible via IPv6 it will automatically connect and deliver using IPv6. If not, it will transparently fall back to IPv4 and continue to deliver as before.