Roller Network Newspipe » Status

Global Crossing is now Level 3

Roller Network — Fri, 07 Oct 2011 03:24:22 +0000

Roller Network has used Global Crossing as a transit provider for our bandwidth mix over the last year. Earlier this year Global Crossing was purchased by Level 3 and we’ve started to receive notices that as of this month our Global Crossing account will now be a Level 3 account. We don’t expect any changes that will impact us operationally (although we’ll be watching closely), and we will be referring to Level 3 as one of our transit providers from this point forward.

Internal Server Maintenance this Weekend

Roller Network — Thu, 29 Sep 2011 05:48:59 +0000

This weekend we’re going to be building a new server for the internal system that runs Asterisk for our office and the Request Tracker ticketing system that handles email for role accounts (like support and billing). Although the current server is still faithfully running with its original Athlon with 786MB of RAM, it’s starting to show its age (RT runs exceptionally slow) and the time has come to give it a forklift upgrade. As such, messages set to support, billing, and sales will be monitored manually (we’re using Mail Mirror on our domain) but they won’t get the usual auto response until after the new server is back online. Our office will also be unavailable by phone during the upgrade, although we rarely receive any calls on the weekends.

The new server we’re building for this upgrade will be an Intel Xeon X3430 with 8GB of DDR3 RAM on a Supermicro X8SIL-F motherboard in a 4U case instead of a 1U since we need to install multiple cards for Asterisk and the drive array controller.

UPDATE: We’ve finished restoring all of the major services and we’re working on the rest of the clean up.

Phone System Bug Fixed

Roller Network — Mon, 05 Sep 2011 21:37:54 +0000

We tracked down and fixed an intermittent bug in our Asterisk dialplan where a call could be dropped when using one of the group dial options (i.e. “press 2 for sales”). The issue was that one of our internal IAX peers was answering and dropping the call if it couldn’t establish a SIP channel rather than sending unavailable/congestion. We’ve altered our dialplan to send congestion explicitly when warranted.

This issue did not affect the 24-hour pager option since it always goes directly to a special voicemail box rather than a group dial, nor were direct dial extensions affected.

Eaton Service Visit

Roller Network — Tue, 23 Aug 2011 15:04:40 +0000

We have an unplanned Eaton service visit today at 17:00 UTC-8. There was an odd behavior with the UPS the resulted in a utility feed side breaker trip, so we’re having them come inspect the equipment, download logs, etc. The result was that the affected UPS ran on battery for about 5 minutes while we investigated and reset the utility feed breaker; there was no impact to our facility operations.

This event is a good example of where batteries have an advantage over a flywheel system. Since the input breaker tripped, the UPS was relying on its DC supply (whether it’s batteries or flywheels) to support the protected load. A flywheel would not have lasted the 5 minutes it took us to check everything. Flywheels, of course, have a lower long term maintenance cost since batteries are consumable items that need to be replaced on a regular basis. The tradeoff is that flywheels do not have the run time endurance of batteries.

UPDATE: No anomalies were found in the UPS performance monitors.

PowerDNS 3.0 Upgrade

Roller Network — Mon, 08 Aug 2011 03:31:13 +0000

As promised, we’re going to be upgrading our PowerDNS authoritative servers from 2.9 to 3.0 in order to address the previously mentioned long TXT records and AXFR support limitations. We will perform the upgrade on Saturday, August 13 starting at 10:00 UTC-8. We expect the upgrade to be transparent.

At that time long TXT record support will go live (they’re already supported in the ACC, just not publishable) and we’ll start working on the front end to manage AXFR. We appreciate your patience.

Year-Round Economizer Cooling

Roller Network — Sat, 06 Aug 2011 11:03:01 +0000

It’s just coming up on 04:00 in the morning here at Roller Network on a Saturday (we’re doing some off-hour line card insertions), and even though it’s summer in our part of the world the high desert climate of Reno still lends itself to suitable economizer cooling mode on our primary facility cooling systems every night almost year-round. Right now it’s about 60F (15C) outside and we’re using that air for cooling rather than run the compressors.

That’s pretty good considering we’re not way up in Alaska, and much better than other competing desert locations at night like Las Vegas (78F/25C) or Phoenix (88F/31C) where the night temperatures are much higher than ours in Reno.

Maintenance Notice: 2011-07-22

Roller Network — Fri, 22 Jul 2011 17:59:41 +0000

We need to swap an array controller card in one of our servers due to excessive “verify fail” errors. We’ve already swapped the drive and the problem persists, so we suspect the controller port may be bad. The affected server handles RT and Asterisk for our office. This maintenance will take place after 18:00 UTC-8)

While the affected server is down the following services will be affected:

Trouble ticket creation (will be queued in the mail spool)
Main office telephones (voice and fax)

We are expecting about 15-30 minutes to unrack the server and replace the array controller. No other services will be affected. This post will be updated as needed.

UPDATE: Completed array card swap successfully. Total time 18 minutes.

Global Crossing IPv6 Trouble

Roller Network — Sat, 05 Mar 2011 19:20:00 +0000

An issue with a customer’s IPv6 access was brought to our attention this morning. After some basic testing, it was apparent the traceroutes were stopping at our Global Crossing PE router (provider side) and we opened a trouble ticket with them to investigate.

A few hours later we discovered that the RIPE website was unreachable. Although the traceroute out indicated it was leaving our network via Sprint, we suspected the return traffic was trying to reach us via Global Crossing. After withdrawing our route announcements to GC we were able to reach RIPE and the customer who origionally brought the issue to our attention.

As this time we are waiting for a resolution on our trouble ticket with Global Crossing regarding this issue.

UPDATE [2011-03-06]: Global Crossing acknowledges that some recent MPLS changes in their network had caused our IPv6 to be null routed. They reverted the most recent changes and connectivity was restored. We will continue to monitor the situation.

Annual UPS Inspection

Roller Network — Fri, 18 Feb 2011 08:00:45 +0000

Eaton will be on site on Wednesday, February 23 at 08:30 local time to perform the annual inspection on our UPS system. No potentially intrusive actions will be taken. If any corrective maintenance is deemed necessary during the inspection, it will be deferred until an appropriate time.

UPDATE: The annual inspection was completed without any noted issues. All battery strings passed load testing.

Christmas 2010 Schedule

Roller Network — Fri, 24 Dec 2010 17:49:59 +0000

From December 23 through December 26 we will be operating on a holiday schedule due to Christmas on Saturday. During this time our technical support staff will be telecommuting from home. If you have a technical support request during this time please open a ticket by email: see the service level comparison chart for expected response times.

The business office will be closed until Monday, December 27.