Outbound Email Spam is teh suck

No mispelling, just playing ‘new internet lingo’ game.  Did I win?

Let’s get serious…

This week, multiple customer accounts were breached.  Starting approximately 3 weeks ago, a phish was sent out that some of our customers responded to, giving out their account information.

We looked through our mail logs and found the users who had been phished and we changed their passwords.

Along the way, we either missed some users who were phished, or another phish was done that we did not detect.

On Monday, 2 accounts that had been phished at some time were used to send spam through our outbound email servers.  By default, our outbound email servers require SASL authentication.  The abusers authenticated to our servers, and over the next couple of hours, we were thoroughly abused, and our servers started slowing down.  Not enough to trigger monitoring, though.  Kudos for performance tuning, spankings for not noticing this until a customer told us.

On Wednesday, we got hit again, by a single account this time, and 18,640 connections later, our servers were again getting exercised.

All this preamble, what is it for, Mike?

I’ll tell you – on Monday our outbound mail servers got onto some of the anti-spam lists, including Yahoo, Hotmail, Comcast.  We did what we could to remove the IPs of our servers from the lists, but Hotmail (in particular) has a 72 hour period for removal.  Ah well.  72 hours does suck, but it is survivable.

Then came Wednesday…and another account was abused, putting us back on those same lists we just got off of, and while still on the Hotmail list, our 72 hours got reset.  Oh that is frustrating.

You see, for many email deployments, requirement of authentication is to significantly reduce the chance of being abused.  For years this worked…

But spammers are smart, and many users do think that it is reasonable that their ISP or email provider would send them email requesting their personal data.   Phishing works because people make the assumption that if the from line lists their ISP, it must be from the ISP.

Unfortunately, that isn’t the case.  A blog post done by an ipHouse employee has some good information and should be read.

So, Mike, what are you going to do about this?

That’s a great question…

I went searching on the tubes for software we could integrate with Postfix, our MTA of choice.

On Wednesday, I implemented software that implements per user limits of number of authentications per hour.  This per hour is a sliding window, as each new authentication is completed, the hour window resets.  This should prevent what we have seen, while allowing most of our users to continue about their email day without impact.

For users using our outbound mail servers in a higher capacity – well we didn’t forget them and have another limit in place that is much higher allowing for pretty decent throughput, but controlled enough that even if something breaks we should not see any issues.

Email itself is not thrown away or even rejected, but it is refused.  Email sent over SMTP has a conversation, and each transaction has a code that is prefixed.  In this case, instead of rejecting via a 500 level message, we return 400 level message – this is a temporary failure and your mail client should retry at a later time.  No, the email isn’t sent when this happens, but it also was not rejected and thrown away.  It is on your computer waiting for a resend.  Just try a little later, maybe an hour later.

If you are a customer who regularly does a lot of email exceeding the first stage limit, please, talk to support, we may have ideas on what you can do to be more efficient, or on a case by case basis, we will move you to another setting.

Email abuse makes for a lack of sleep.  Trained professionals should not try this at home without some coffee.