Roller NetworkWe had to switch responses for “inboundproxy.com” from 4xx to 5xx on our Outbound Mail (smarthost/submission) server this morning due to a massive flood of attempts to the domain causing inode exhaustion on /tmp. The domain “inboundproxy.com” does not exist in DNS, which is normally a 4xx response.
This Saturday, February 1st during our recurring routine maintenance window (between 10:00 and 22:00) we will be physically relocating a server that handles Outbound Mail service (also known as smtpauth.rollernet.us). The expected duration is 1 to 10 minutes, or approximately the time it will take to shutdown, relocate, and restart. Replacement hardware will be brought online in parallel in preparation for an upgrade at a later date.]
UPDATE: This maintenance has been completed.
EVENT: NV Energy Planned Outage: 2013-11-23 @ 06:30
DATE: Saturday, November 23, 2013
TIME: 06:30 to 08:00 local time (US/Pacific)
SERVICE IMPACT: None expected.
DETAILS:
We’ve received notice that NV Energy will be performing line work in our area with a planned outage on Saturday, November 23 between 6:30am and 8:00am local time (US/Pacific). Roller Network staff will be on site beginning at 6:00am to monitor as needed.
This event will be treated as a substitute to our bi-weekly recurring 09:00 automatic generator exercise/test. Updates will be posted to online to our Twitter feed and this post on the Newspipe.
No impact to Roller Network services is expected.
UPDATES:
06:44 – Power off, generator on. All is normal.
08:04 – Power restored. The system will now wait briefly before transferring back.
08:27 – This event has completed. No service impact, as expected.
What: Service interruption to microwave and WiMAX services through Red Peak due to UPS battery exhaustion.
Where: Transmission tower in Sun Valley, NV
Duration of impact: August 8, 2013 20:02:47 to 20:27:09 local time.
At 19:25:11 local time on August 8, 2013 a lightning strike at or near an NV Energy substation on Valley Road caused a widespread utility power outage in the north valleys, including Sun Valley and the transmitter towers on Red Peak where Roller Network leases space to operate licensed microwave transmitters and our WiMAX base station. At 19:25:27 we were notified of an emergency “on battery” condition.
Roller Network maintains a UPS to provide power until the site’s permanent automatic generator starts to provide emergency power, but our equipment remained on battery. Our procedure for this site is to deploy a portable generator of our own if we do not see a power restoration from the site generator within 30 seconds, and so the portable generator was deployed immediately. We were already proactively monitoring our extended network due to weather.
Unfortunately our UPS batteries were exhausted at 20:02:47 as we were heading up Clear Acre to the access road in Sun Valley (intersection of 1st and Sun Valley). We arrived to find the site dark; we were told the site generator had a dead starting battery. We deployed our portable generator with extension cords into the shelter building and restored power to our UPS at 20:27:09.
We relied on our UPS for power up to 30 minutes, however in the event of a site generator fail-to-start (which is not ours to maintain) this proved to be insufficient time to deploy a portable generator. Travel to the site was delayed due to traffic signal outages and potentially life-threatening lightning conditions on the mountain top.
In response Roller Network will be deploying additional battery packs to provide more time to allow for dangerous site conditions were automatic generators not maintained by Roller Network exist to provide more buffer time to respond on our own.
Sites we operate without automatic generators already have sufficient battery packs for at least 4-5 hours of UPS runtime. Our procedure for these sites is to deploy portable generators after three hours on battery. Sites with automatic generators not maintained by Roller Network will now be treated the same as sites without generators.
No equipment was damaged nor was any manual intervention required beyond connecting the portable generator.
Customers with a service level agreement should submit requests for a service credit, if desired, within 30 days.
August 8, 2013
19:25:11 – Loss of utility power from NV Energy.
19:25:27 – Initial emergency notification.
19:26:00 – Portable generator deployment following emergency state.
20:02:47 – UPS battery exhaustion.
20:27:09 – Restored power with portable generator.
21:53:31 – Utility power restored.
After 8 years of having Sprint as part of our AS transit mix we will be removing our circuits to AS1239 on February 4th, 2013. While it’s been our most reliable, outlasting other providers that have come and gone, it’s far too overpriced for us to seriously consider keeping it. In its place we have recently turned up BGP with Charter Communications and have another Ethernet circuit from AT&T pending turn up (additional equipment was recently placed in AT&T’s fiber bay in our telco room). We’ve chosen to add these two providers to our local mix for not only cost savings, but because these providers are also large players in our region. Having direct connections to them will allow us to keep more traffic in region for our local customer base. Both also support native IPv6 and meet our transit provider parity mandates.
Effective today, January 18, 2013, we have lowered the local preference in our AS below transit default for all routes received from AS1239 and are tagging our announcements with the lowest supported local preference community. These actions will divert almost all traffic transiting through Sprint in favor of others. In two weeks we will administratively disable all BGP neighbors with AS1239 and remove the physical circuits.
We do not anticipate any impact by removing Sprint, although you may notice new hops in traceroutes. Making these changes to our routing policy ahead of time allow us to make such a transition without impact, and is a perfect example of how being autonomous allows us to add and remove providers at will. We do not make such decisions based on price alone and frequently pay reasonable premiums for critical services, the pricing associated with Sprint was over four times as expensive as the nearest alternative with no upgrade path through Sprint.
As always, we appreciate the support of our customers and look forward to serving you in 2013 and beyond.