For reasons unknown at this time, it appears that MX mail.rollernet.us was refusing connections on port 25. We have since corrected the problem by restarting all potentially affected Postfix instances.
Because to this, we received several reports from customers who only had a single MX record configured. This is not a supported configuration. By using both MX records, they are designed to compliment each other in the event a fault in one prevents it from processing incoming mail. The two pools are unique, making it highly unlikely that a fault in one will affect the other. This is not a strict requirement in that we’ll refuse to help, but it is to your benefit to configure both. (We will refuse requests for outage credits, however, because it is in the help docs to use both.)
In order to maintain the highest level of service possible, please ensure that you are using both MX “mail” and “mail2” MX records. Customers that have both did not experience any interruptions.
[UPDATE] The cause is believed to be memory exhaustion caused by a debugging statement that was accidentally left in. This was a procedural error on our part as the debug should have been removed when the trouble ticket it was associated with closed.
We have no idea why there is such a lack of progress with Verizon, so we ordered a short term 6M multilink T1 circuit with Global Crossing as a band-aid. While we are keenly aware that it has no hope of fully taking over from a complete failure of our high capacity circuits, the idea is at least we won’t be completely off the air. It also brings something important to the table on its own: native IPv6. Global Crossing, while one of the more expensive providers, is well known for IPv6 support.