Month: October 2009

Categories
Status

Move (mostly) Complete

We successfully moved the entire facility over the weekend with a few minor hiccups with hosted mail service that didn’t show up during testing. In any case, everything is back to normal as of Monday morning. There’s still a lot of detail work to do and everything else (like office furniture) still needs to make its way over to the new place, but at this point there is nothing left to move that could impact our normal operations.

There, however, is one negative aspect to report: because Verizon is still unable to provide service to us we are running single-homed on our Sprint service. This should not be a problem as long as Sprint doesn’t have any outages, however, this is not something we’re comfortable with. We’ve tried to make it clear to Verizon that no matter what their policy is doesn’t change the fact that we need the service and someone will provide it.

Categories
Uncategorized

New Facility Progress #10

We’re on the eve of the move and this is probably the last new facility update before actually moving in. Verizon is still missing in action, but we’re going ahead without them since it’s clear that it isn’t getting resolved anytime soon. We would like to thank Sprint for being patient with us since they were ready to hot cut back in August and we kept delaying to try to resolve things with Verizon.

PA060006-alt

Final layout of a UPS and its battery cabinet before wiring. Three of these complete units will fit into our UPS electrical room (up to 4 supported) to act as one parallel redundant unit.

PA070005

A close up of the UPS wiring. The cable is 1/0 AWG.

DCP_2605

Some of the overhead piping in the UPS electrical room.

PA070006

One of the two distribution panels for the server room (right) and the UPS paralleling panelboard (left).

PA070001

Overhead piping between the UPS electrical room to the panelboard that supports it. The panelboard is connected to our generator automatic transfer switch allowing it to be fed by utility power or generator power. As you can see, this is in the warehouse portion of the facility, which will house the generator.

DCP_2600

The original load center (left), the new panelboard for our stuff (middle), and our automatic generator transfer switch (right).

DCP_2608

The panelboard with its dead front installed, wiring cleaned up, and the transfer switch after turning the power on for the first time. The generator will be installed within the week and piped to the connectors on top of the transfer switch. Wiring the generator is as simple as connecting it to the “emergency” contacts inside the transfer switch after the not so simple process of moving a very heavy diesel engine.

The next update will be after the major move of circuits and equipment this weekend. There’s still plenty for us to do before we can consider it done and take a break, so stay tuned!

Categories
Announcements Status

Roller Network Facility Move Notice

The time for us to move our facility has come. We originally planned to move the last week of August in order to open in October, but if you’re been following our story with Verizon you already know why that didn’t work out. The Verizon circuit was to be used to allow a minimal disruption outage by tunneling both sites. However, they are still not following through, and we can’t wait forever.

As such, we will be performing a hot-cut of our Sprint service on Friday, October 16, 2009 at 14:00 US/Pacific time (UTC-8). All traffic will be rerouted over our lower capacity SAVVIS circuit beginning at 13:30 UTC-8. We will begin announcing our address space via Sprint at the new location once at least 50% of the equipment has been moved and stabilized. We have timed the hot-cut to correspond with the beginning of the weekend where our average utilization falls within what the SAVVIS circuit can handle, although the slowdown may still be noticeable. Unfortunately this unavoidable due to the Verizon trouble.

THERE WILL NOT BE A COMPLETE OUTAGE. Groups of equipment are staged so that their backup/secondary remains online while the primary is moved. Once the primary is moved the secondary will follow. You may notice some brief hiccups as traffic is rerouted between locations, but you should not expect any substantial outages. Non-critical services (such as our company website and the account control center) do not have redundant counterparts and will be offline for 15 to 30 minutes while the servers are moved. Critical services (mail services, hosted mail, DNS services) will not be interrupted.

Because the move will affect our network in a substantial way we have moved our status page to an offsite location served by third-party DNS. You can follow the move progress online at:

www.rollernetstatus.com

This new status page will be permanently replacing the old status page at status.rollernet.us. If you have bookmarked the old status page, please update your bookmarks.

Our goal is to ensure that the move is as painless and transparent as possible. If you have any questions or concerns please feel free to contact us. Further updates will be posted on the newspipe (www.rollernet.us/wordpress) and status page (rollernetstatus.com) as we progress.

Categories
IPv6

Verizon Update: No Contact Yet

We had a promising conference call with Verizon on Tuesday and a couple emails from Verizon representatives (who deferred to our assigned account team, whoever that may be, we don’t know), but unfortunately nobody who can officially help has yet to contact us as promised and no further progress has been made as of this post.

For us this is merely a continuation of the same problem we had in August/September when our original account manager disappeared.

Categories
IPv6

Verizon Post-Slashdot Followup

To Verizon’s credit, we received a response back from Verizon that they’re going to escalate our issue to network engineering. We aren’t sure where it will go from there. In the meantime, there were several questions in the discussion that we should post as a followup.

You can’t announce less than a /32 in IPv6. It was designed to aggregate everything into /32’s.

That used to be true. ARIN (2620:0::/23), AFRINIC (2001:43F8::/29), and APNIC (2001:0DF0::/29) all allow for PI IPv6 to fill the gap that IPv6 left by excluding (sane, timely, managable) multihoming in the design of IPv6. RIPE decided to open the door to PI IPv6 at RIPE58. Multihoming was (is?) is a big point of contention in the community. The future of BGP may end up being a lookup-based system to reduce the overhead of carrying a full table.

What does IPv6 offer you? Why do you care?

Roller Network is not new to IPv6. We already offer IPv6 enabled services and have customers that use it, so yes, it’s important for us to keep the same level of service at the very least. No, it’s not a huge amount of traffic, but someday it might matter, and we want to be ahead of the game rather than be short sighted and scramble at the last minute. Adoption is also a classic “chicken and the egg” problem so we’re trying to do our part by providing IPv6 enabled services.

Verizon is well known for not providing IPv6. What happened?

This is true. However, we did fully disclose our routing intentions and service requirements in May when we started all of this, and they confirmed it with the order so we were led to believe it was not a problem. However, the lost the order somewhere between May and August and turned it up with static IPv4 only. The implementations manager that processed the original order also went missing during that time and as of September our account manager was no longer with the company. Of course we forwarded copies of everything we had and it was re-engineered to a different city that supported IPv6. During all of this was bi-weekly calls for status updates, trying to find our new account manager, etc. That brings us to now.

Why don’t you just leave Verizon?

We’ve put a lot of effort into this (it’s fiber) and we want to try to work through the myriad of bureaucracy to change things for the better as long as someone from Verizon is willing to work with us. It’s not just affecting us and our customers, but IPv6 adoption as a whole. A little publicity to an issue that has  generally been kept quiet in the past might help. It could also hurt, but others have come before us quietly, so we’ll have to take a chance.

Is their router filtering IPv6?

No. They’re not carrying routes in BGP most likely because it’s not allowed in their prefix lists. This is on an OC-12 delivered as Ethernet, not a residential connection. We wouldn’t have made such a big deal over this otherwise.

Verizon is right, you’re wrong.

As mentioned before, we’re already getting the same service from other providers like Sprint. Even Hurricane Electric’s free tunnel broker supports IPv6 PI and BGP peering. We’re already visible in looking glasses worldwide. Verizon may have been right years ago prior to at least October 2006 when ARIN started issuing PI IPv6, but they’re not keeping up with where the industry is going. Again, the lack of a usable multihoming mechanism in IPv6 forced the issue.

Your WordPress got shashdotted and I couldn’t read the article!

Yeah, sorry about that. (And you actually tried to RTFA?) We weren’t expecting front page coverage. Our web server is very low traffic and mostly serves static pages. It’s a 700MHz PIII. However, it’s moving along just fine after adding the WP Super Cache plugin which provides static caching. Default wordpress really sucks performance-wise.

Now what?

We’ll keep posting updates as they happen. If you’d like to keep following our Verizon saga, have an interest in IPv6, or both, please subscribe to our RSS feed. We’ve added an IPv6 category as well. If by some miracle Verizon moves in our favor we’ll try to post a followup story to Slashdot.