[M5Hosting] M5 Hosting News, Events and Policy Changes

[Michael J McCafferty]

Dear M5 Hosting Customer

             We have just passed the Thanksgiving Day holiday (U.S. 
observance). This is the time when we give thanks for the bounties we 
have been so lucky to enjoy. This year M5 Hosting has so very much to 
be thankful for.
             While we have been hosting dedicated servers and web 
sites for much longer, this year has been a year of tremendous growth 
for us. We have had brilliant luck to have had the opportunity to 
earn your trust and continued patronage this year. It has been 
rewarding and good fun working with some of you on special projects, 
and working for all of you as your host. These are some of the things 
for which we give thanks, you our customers, and our success... along 
with our health, our friends and our families.

             These bounties have not come without challenges. I'd 
like to share with you the challenges and what we are doing about them:

1)      On 11/16/06, a cross-connect between our main network rack 
and one of our dedicated server racks became disconnected.
         All of our racks make a home-run back to the data center's 
"meet me room". The racks that are connected together are cross 
connected in the meet me room. The meet me room is supposed to be a 
low traffic, controlled environment. It turns out that one of the 
Data Center's engineers was working on a connection near ours, and 
may have inadvertently disconnected one of our cross connections. 
This caused one of our racks to lose connectivity for about 1hr. This 
affected 42 dedicated server customers. Those who were affected 
received Nagios alerts at the time of the outage. Those who sent 
email to support received updates during and after the incident.
         We are ordering redundant cross connections between all of 
our racks and our main network rack. The connections will be 
configured to be redundant and fault-tolerant. In the event that a 
single cross-connect fails again, the affected rack will not lose 
connectivity. This will be deployed within the next month. Look for a 
maintenance announcement.

2)      On 11/20/06 beginning at approximately 12:30am (PDT), a very 
new customer experienced a denial of service attack which exhausted 
much of our available bandwidth to most of our customers. The attack 
was quenched in about 1hr. During the 1hr time period, affected 
customers may have experienced high packet loss, high latency and/or 
loss of connectivity. Denial of service attacks have been a challenge 
for us and our customers recently. They are a common challenge on the 
Internet. While they are common, they can be devastating.
         In every case of a DoS attack on our network over years, the 
attacks have been related to IRC (Internet Relay Chat). In most cases 
it was traced to a customer operating an IRC server on their server 
and a user logged in to that server attacked the server or a user on 
the server visiting an IRC channel on another server and drawing the 
ire of an attacker. In at least two other cases our customer suffered 
a security compromise to their server and the customer server was 
used as part of an attack, or was attacked because it was used in an 
attack on yet another server. Again, all of this related to IRC. So 
far, we have not filtered any port or protocol at our border 
firewalls, except when specifically requested by you, for your 
server. However, this policy has changed.
         In response, we have implemented firewall rules which will 
prevent IRC traffic to or from our network except for customers whom 
we know are using it currently or who specifically ask for IRC to be 
allowed to/from their server. Before implementing the firewall rules, 
we sniffed the network for a short period to determine a list of our 
customers who are using IRC. We will confirm that those customers are 
intentionally using IRC. If the new rules have blocked your IRC 
traffic and you want it to be turned back on, please send an email to 
support and we will add you to the exclusion list.
         This is an interim solution. We will create a permanent 
policy to address the risks of IRC, to our network, your server 
connectivity. If you have suggestions or comments regarding an IRC 
specific policy, please send them to us via email. The goal of any 
policy will be to protect your server as well as our other customers 
from DoS attacks and other security risks associated with IRC. Most 
dedicated server hosts do not allow IRC at all. Until this new policy 
is created, we will not open IRC for any new customers who sign up 
after today and the default policy to block standard IRC ports by 
firewall rules.
We will greatly welcome any suggestions you may have regarding a new 
IRC policy.

         As always, your feedback is encouraged, both on this message 
and the actions and events it describes. We send these messages from 
time to time for the purpose of clearly and honestly keeping you 
informed on issues which relate to the service you trust us to 
provide. Some hosts will down-play or deny their challenges. I think 
you will appreciate our policy of openly and honestly sharing the 
details of ours. We want you to know what we are doing to improve and 
to be fully aware of any operational issues which affect your 
business or project.

         Thank you for your continued trust !

Sincerely,
Mike


************************************************************
Michael J. McCafferty
Principal, Security Engineer
M5 Hosting
http://www.m5hosting.com

You can have your own custom Dedicated Server up and running today !
RedHat Enterprise, CentOS, Fedora, Debian, OpenBSD, FreeBSD, and more
************************************************************