Home > News >
Large-Scale Packet Loss [ServePath Data Centre]
Large-Scale Packet Loss [ServePath Data Centre]
by John Keagy, President ServePath, 26 February 2003
ServePath experienced a network incident this morning that caused large-scale packet loss. We had a customer experiencing an attack on switch 19 and an error that we made in diagnosing the attack prolonged the situation. New training and documentation procedures are being implemented to avoid this in the future.
We are nearly half way through a list of projects that will improve network resilience and our ability to detect and remedy problems:
-
VLANs: This project will give every customer their own virtual network so that they are immune to broadcast storms from other customers. This is critical for withstanding distributed denial of service attacks, syn floods, and other high traffic scenarios. The project is delayed but will be complete within 48 hours.
-
Automatic network healing: We have a new tool under development that will take action automatically to defend against network attacks. Currently our responses require substantial manual efforts that are prone to delays and errors.
-
Staffing: We've been under-staffed due to our fast growth. We are excited to report that we have many quality new people in training this week that will greatly improve our support services, especially after-hours.
-
New network expansion: We will soon be announcing some powerful new peering and transit arrangements.
-
Communication: We need to do a better job communicating with our customers. This will be an ongoing process but we are developing plans and we will keep you apprised of our progress.
- Traffic analysis: We have deployed a system that is gathering data on the origination and destination of our IP traffic. This data will be used to optimize network performance and select peering partners and transit vendors.
Thank you for your business and understanding.