Categories
Status

Redundant Switch Fault

On Saturday evening (September 3, 2009) we experienced a short general outage caused by our DNS and SQL cache pool servers rebooting. Normally this wouldn’t be a problem, but the PowerDNS Recursor package didn’t start at boot time, and a lack of DNS meant a general lack of anything useful taking place.

It was odd that some equipment rebooted itself while others didn’t, so after a lot of thinking, we decided that one of the APC 30A redundant switches we use was probably faulted because this isn’t the first time we’ve seen this. We pulled the switch from service (causing another reboot – although if we were right we didn’t want it risking the system anymore) and opened the cover. Inside we found relay contacts with pitting and arching:

DCP_2588 DCP_2590 DCP_2591

The other relays we opened up weren’t nearly as bad, but they still exhibited discoloration and pitting on the contacts. The one in the pictures was loaded between 10 and 15 amps and it’s supposed to be rated for 30 (or 24 derated). Because this is the second failure we’ve had with this device we’ve decided to remove the remaining ones from service as they are likely to suffer the same fate in the future.

We apologize for the recent bumps in the normally smooth operation you’ve come to expect from us. We understand that your mail and DNS service is important to you, and to us, since we use the same services for our mail. As such, a discount/credit will be forthcoming.

Categories
Announcements New Facility News Status

Phone Service Unavailable Beginning Sept. 29

Our phone lines are scheduled to be moved to our new office the afternoon of September 29. However, the Verizon circuit has still not been re-provisioned correctly and we don’t have any IP service. Because of this we can’t move our VOIP concentrator to the new office and we will be unable to take calls to our main number (775-284-0383) until further notice.

We will set up a temporary system with some old hardware to answer the lines and take voice messages as a workaround.

UPDATE 10-20-2009: Phone services have been restored.

Categories
Announcements New Facility News Status

Development Freeze

We’re currently in the build out phase for our new facility. Eventually this means physically moving and reassembling the service in a new location. In order to ensure the transition is as smooth as possible, we are freezing all development on new features, requests, and changes to the service. Not only are all of our staff involved in the physical aspect of the build out, but we feel that it’s of critical importance that our services be kept in a known stable state during this time to ensure the quality of service you’ve come to expect from us. The task of server room and network assembly is one that must be done by our own staff to ensure it meets our specifications without delays. Moving an operation like ours without interrupting services is a monumental undertaking – one made easier by ensuring that the system is kept in a stable state.

We apologize if this causes any inconveniences. This kind of move is not a common occurrence, and rest assured, we’ll be back and better than before after we’re done.

Categories
Status

System Maintenance Notice, ID#340

DCP_2484

This is just a reminder that tomorrow (July 11, 2009) between 11:00 and 12:00 local time (UTC-8) we will be performing system maintenance on our master database server. During this time the account control center will be unavailable. No other services will be affected. For complete details and updates as it progresses, please see System Maintenance Notice, ID#340 on the forums. We will be adding an HP StorageWorks MSA direct attached array and E500 controller with 15k SAS drives to increase the table space storage capacity.