Categories
Announcements Status

Roller Network Facility Move Notice

The time for us to move our facility has come. We originally planned to move the last week of August in order to open in October, but if you’re been following our story with Verizon you already know why that didn’t work out. The Verizon circuit was to be used to allow a minimal disruption outage by tunneling both sites. However, they are still not following through, and we can’t wait forever.

As such, we will be performing a hot-cut of our Sprint service on Friday, October 16, 2009 at 14:00 US/Pacific time (UTC-8). All traffic will be rerouted over our lower capacity SAVVIS circuit beginning at 13:30 UTC-8. We will begin announcing our address space via Sprint at the new location once at least 50% of the equipment has been moved and stabilized. We have timed the hot-cut to correspond with the beginning of the weekend where our average utilization falls within what the SAVVIS circuit can handle, although the slowdown may still be noticeable. Unfortunately this unavoidable due to the Verizon trouble.

THERE WILL NOT BE A COMPLETE OUTAGE. Groups of equipment are staged so that their backup/secondary remains online while the primary is moved. Once the primary is moved the secondary will follow. You may notice some brief hiccups as traffic is rerouted between locations, but you should not expect any substantial outages. Non-critical services (such as our company website and the account control center) do not have redundant counterparts and will be offline for 15 to 30 minutes while the servers are moved. Critical services (mail services, hosted mail, DNS services) will not be interrupted.

Because the move will affect our network in a substantial way we have moved our status page to an offsite location served by third-party DNS. You can follow the move progress online at:

www.rollernetstatus.com

This new status page will be permanently replacing the old status page at status.rollernet.us. If you have bookmarked the old status page, please update your bookmarks.

Our goal is to ensure that the move is as painless and transparent as possible. If you have any questions or concerns please feel free to contact us. Further updates will be posted on the newspipe (www.rollernet.us/wordpress) and status page (rollernetstatus.com) as we progress.

Categories
IPv6

Verizon Update: No Contact Yet

We had a promising conference call with Verizon on Tuesday and a couple emails from Verizon representatives (who deferred to our assigned account team, whoever that may be, we don’t know), but unfortunately nobody who can officially help has yet to contact us as promised and no further progress has been made as of this post.

For us this is merely a continuation of the same problem we had in August/September when our original account manager disappeared.

Categories
IPv6

Verizon Post-Slashdot Followup

To Verizon’s credit, we received a response back from Verizon that they’re going to escalate our issue to network engineering. We aren’t sure where it will go from there. In the meantime, there were several questions in the discussion that we should post as a followup.

You can’t announce less than a /32 in IPv6. It was designed to aggregate everything into /32’s.

That used to be true. ARIN (2620:0::/23), AFRINIC (2001:43F8::/29), and APNIC (2001:0DF0::/29) all allow for PI IPv6 to fill the gap that IPv6 left by excluding (sane, timely, managable) multihoming in the design of IPv6. RIPE decided to open the door to PI IPv6 at RIPE58. Multihoming was (is?) is a big point of contention in the community. The future of BGP may end up being a lookup-based system to reduce the overhead of carrying a full table.

What does IPv6 offer you? Why do you care?

Roller Network is not new to IPv6. We already offer IPv6 enabled services and have customers that use it, so yes, it’s important for us to keep the same level of service at the very least. No, it’s not a huge amount of traffic, but someday it might matter, and we want to be ahead of the game rather than be short sighted and scramble at the last minute. Adoption is also a classic “chicken and the egg” problem so we’re trying to do our part by providing IPv6 enabled services.

Verizon is well known for not providing IPv6. What happened?

This is true. However, we did fully disclose our routing intentions and service requirements in May when we started all of this, and they confirmed it with the order so we were led to believe it was not a problem. However, the lost the order somewhere between May and August and turned it up with static IPv4 only. The implementations manager that processed the original order also went missing during that time and as of September our account manager was no longer with the company. Of course we forwarded copies of everything we had and it was re-engineered to a different city that supported IPv6. During all of this was bi-weekly calls for status updates, trying to find our new account manager, etc. That brings us to now.

Why don’t you just leave Verizon?

We’ve put a lot of effort into this (it’s fiber) and we want to try to work through the myriad of bureaucracy to change things for the better as long as someone from Verizon is willing to work with us. It’s not just affecting us and our customers, but IPv6 adoption as a whole. A little publicity to an issue that has  generally been kept quiet in the past might help. It could also hurt, but others have come before us quietly, so we’ll have to take a chance.

Is their router filtering IPv6?

No. They’re not carrying routes in BGP most likely because it’s not allowed in their prefix lists. This is on an OC-12 delivered as Ethernet, not a residential connection. We wouldn’t have made such a big deal over this otherwise.

Verizon is right, you’re wrong.

As mentioned before, we’re already getting the same service from other providers like Sprint. Even Hurricane Electric’s free tunnel broker supports IPv6 PI and BGP peering. We’re already visible in looking glasses worldwide. Verizon may have been right years ago prior to at least October 2006 when ARIN started issuing PI IPv6, but they’re not keeping up with where the industry is going. Again, the lack of a usable multihoming mechanism in IPv6 forced the issue.

Your WordPress got shashdotted and I couldn’t read the article!

Yeah, sorry about that. (And you actually tried to RTFA?) We weren’t expecting front page coverage. Our web server is very low traffic and mostly serves static pages. It’s a 700MHz PIII. However, it’s moving along just fine after adding the WP Super Cache plugin which provides static caching. Default wordpress really sucks performance-wise.

Now what?

We’ll keep posting updates as they happen. If you’d like to keep following our Verizon saga, have an interest in IPv6, or both, please subscribe to our RSS feed. We’ve added an IPv6 category as well. If by some miracle Verizon moves in our favor we’ll try to post a followup story to Slashdot.

Categories
Status

Slashdotted!

We were slashdotted today after our woeful story of Verizon made the front page. A basic WordPress install is very bad on server performance, and the fact that our public facing pages are all static means our web server wasn’t quite up to the task. But after some tuning and static caching, we’re in better shape.

Technology: Verizon Refuses To Provide Complete IPv6

Slashdot is welcome here; it’s a rare honor to be taken down by a front page link. 😉

Categories
Status

Redundant Switch Fault

On Saturday evening (September 3, 2009) we experienced a short general outage caused by our DNS and SQL cache pool servers rebooting. Normally this wouldn’t be a problem, but the PowerDNS Recursor package didn’t start at boot time, and a lack of DNS meant a general lack of anything useful taking place.

It was odd that some equipment rebooted itself while others didn’t, so after a lot of thinking, we decided that one of the APC 30A redundant switches we use was probably faulted because this isn’t the first time we’ve seen this. We pulled the switch from service (causing another reboot – although if we were right we didn’t want it risking the system anymore) and opened the cover. Inside we found relay contacts with pitting and arching:

DCP_2588 DCP_2590 DCP_2591

The other relays we opened up weren’t nearly as bad, but they still exhibited discoloration and pitting on the contacts. The one in the pictures was loaded between 10 and 15 amps and it’s supposed to be rated for 30 (or 24 derated). Because this is the second failure we’ve had with this device we’ve decided to remove the remaining ones from service as they are likely to suffer the same fate in the future.

We apologize for the recent bumps in the normally smooth operation you’ve come to expect from us. We understand that your mail and DNS service is important to you, and to us, since we use the same services for our mail. As such, a discount/credit will be forthcoming.