Outbound Email Spam is teh suck

No mispelling, just playing ‘new internet lingo’ game.  Did I win?

Let’s get serious…

This week, multiple customer accounts were breached.  Starting approximately 3 weeks ago, a phish was sent out that some of our customers responded to, giving out their account information.

We looked through our mail logs and found the users who had been phished and we changed their passwords.

Along the way, we either missed some users who were phished, or another phish was done that we did not detect.

On Monday, 2 accounts that had been phished at some time were used to send spam through our outbound email servers.  By default, our outbound email servers require SASL authentication.  The abusers authenticated to our servers, and over the next couple of hours, we were thoroughly abused, and our servers started slowing down.  Not enough to trigger monitoring, though.  Kudos for performance tuning, spankings for not noticing this until a customer told us.

On Wednesday, we got hit again, by a single account this time, and 18,640 connections later, our servers were again getting exercised.

All this preamble, what is it for, Mike?

I’ll tell you – on Monday our outbound mail servers got onto some of the anti-spam lists, including Yahoo, Hotmail, Comcast.  We did what we could to remove the IPs of our servers from the lists, but Hotmail (in particular) has a 72 hour period for removal.  Ah well.  72 hours does suck, but it is survivable.

Then came Wednesday…and another account was abused, putting us back on those same lists we just got off of, and while still on the Hotmail list, our 72 hours got reset.  Oh that is frustrating.

You see, for many email deployments, requirement of authentication is to significantly reduce the chance of being abused.  For years this worked…

But spammers are smart, and many users do think that it is reasonable that their ISP or email provider would send them email requesting their personal data.   Phishing works because people make the assumption that if the from line lists their ISP, it must be from the ISP.

Unfortunately, that isn’t the case.  A blog post done by an ipHouse employee has some good information and should be read.

So, Mike, what are you going to do about this?

That’s a great question…

I went searching on the tubes for software we could integrate with Postfix, our MTA of choice.

On Wednesday, I implemented software that implements per user limits of number of authentications per hour.  This per hour is a sliding window, as each new authentication is completed, the hour window resets.  This should prevent what we have seen, while allowing most of our users to continue about their email day without impact.

For users using our outbound mail servers in a higher capacity – well we didn’t forget them and have another limit in place that is much higher allowing for pretty decent throughput, but controlled enough that even if something breaks we should not see any issues.

Email itself is not thrown away or even rejected, but it is refused.  Email sent over SMTP has a conversation, and each transaction has a code that is prefixed.  In this case, instead of rejecting via a 500 level message, we return 400 level message – this is a temporary failure and your mail client should retry at a later time.  No, the email isn’t sent when this happens, but it also was not rejected and thrown away.  It is on your computer waiting for a resend.  Just try a little later, maybe an hour later.

If you are a customer who regularly does a lot of email exceeding the first stage limit, please, talk to support, we may have ideas on what you can do to be more efficient, or on a case by case basis, we will move you to another setting.

Email abuse makes for a lack of sleep.  Trained professionals should not try this at home without some coffee.

2 Replies to “Outbound Email Spam is teh suck”

  • Hey Mike,

    We have a serious new Postfix system, and we had the same exact Nigerian nightmare as you write about here.

    Yeah, it’s teh suck all rights!!

    Nigerians were using stolen credit cards to sign up for accounts, and so far our newb-scribers have smartened up and aren’t sending their id’s/ps’s to them anymore. So far this month anywaze. We stopped the online sign-upability, but we still want to impose limits for the next inevitable phishing incident.

    > On Wednesday, I implemented software that implements per user limits of number of
    > authentications per hour.

    So what change did you made? We’re looking to do exactly the same thing. I’m trying to find the specifics for my linux guy.

    We’re also looking at running this milter, that counts outbound failures:


    Cyber Mesa Telecom

  • @dannyclyde Hey there!

    Implemented parts of policyd (cluebringer or cbpolicyd). (hate when kiddies decide to name things)

    Set it up for a sliding 1 hour window and set metrics, and allow customers to go over by contacting us and putting them into a different bucket.

    Has worked well so far. Since we only allow SASL authenticated relay – I set the authentication key to be their SASL username – what IP they come from doesn’t matter, so even if an account is breached and abuse is coming from multiple IP addresses – it is caught.

    Best of luck!


Comments are closed.