I said it, you read it, hope you aren’t offended.
ipHouse has 3 1Gbps links to the Internet connected to two routers and from there enters our core network at gigabits per second speed.
Each customer aggregation L2/L3 switch is dual-linked with 2 1Gbps links though these boxes can only take so much traffic.
The destination for the denial of service is a customer with a 1Gbps port *limited* to 10Mbps so all of that traffic was traversing our network then being dropped at the destination port.
This took a while to find as we have around 2000 destinations for traffic (probably higher, I’m just guessing). Doug and I were on the prowl, but this is my experience from the evening.
I needed to find the destination L2/L3 switch then from there find the destination for said traffic. Using our trusty LogicMonitor interface, I was able to look at the core networking and find the destination customer aggregation switch. What a time-saver that was!
This particular L2/L3 switch has 98 1Gbps ports and over 130 VLANs and I had to go through them one at a time to see which ports/VLANs were taking the traffic.
I actually missed this one as I went through because 10Mbps of use isn’t normally a trigger for me.
As I went back through I then noticed it was a flatlined port for traffic so I pulled up the history and there it was. This was the destination I was looking for.
I did 90% of this without a computer which took a long time (I couldn’t get to a system at that time) and it was slow. I should have just gotten on a real system to continue my investigation. I mean, yes, LogicMonitor works via an iPad, works well really, but it is no replacement for a real computer.
Less than 10 minutes after getting to a real computer I found it.
Less than 10 minutes after that (remember, I am coming in the same way others are, like you, with all the latency and dropped packets) I had the customer null-routed (no longer on the Internet) and everything recovered. When I say recovered, I mean traffic was flowing and we were dropping a lot less packets. For some external connectivity, though, it was still pretty slow and/or unusable.
Being pedantic: ipHouse wasn’t actually down but with the very high packet loss, well, it sure seemed that way to most everyone.
Sorry you had to deal with this.