SPAM fighting is affecting legitimate eMail

People doing irresponsible things in fighting SPAM are conforming to the definition of "Idiots at Work". The Google Postmasters are therefor in this group.

But they are not alone. The vast majority of Postmasters out there are similar acting inresponsible when it comes to SPAM.

Usually I am not affected by those majority of wrong Anti-SPAM-procedures, however as a GMail user I am in serious trouble, as it is no more possible for me to forward legitimate eMail to my GMail account.

The facts are simple but neglected by nearly all Postmasters out there:

  • Blocking SPAM is not blocking SPAM.
  • Blocking SPAM is only harming legitimate eMail.
If you are puzzled, here is one proof (there are zillions out there).

Google is lying

Gmail's MTAs are broken because of Anti-SPAM-procedures. And they are lying.

Here is the message:
# telnet gmail-smtp-in.l.google.com. smtp
Trying 209.85.129.27...
Connected to gmail-smtp-in.l.google.com..
Escape character is '^]'.
421-4.7.0 [217.160.78.171] Our system has detected an unusual amount
421-4.7.0 of unsolicited mail originating from your IP address. To protect
421-4.7.0 our users from spam, mail sent from your IP address has been
421-4.7.0 temporarily blocked. Please visit http://www.google.com/mail/help/bul
421 4.7.0 k_mail.html to review our Bulk Email Senders Guidelines. k29si11431419fkk.5

This message is wrong!

All my MTA was doing is to send a backlog of some 100 undeliverables (per minute, because of From abuse) to my Gmail account. At least it was trying to do. However this did not work, as Google always stops delivery after 100 messages or so.

Following are the facts:

  • The messages are no SPAM: They are the result of SPAM, but they itself are no SPAM. The SPAMmed MTAs must deliver this message. This is an RFC requirement! You cannot expect an RFC requirement to be counted as SPAM (else RFCs would require SPAM).
  • The messages are not unsolicited: I want this messages to be delivered to my GMail account (to let GMail filter them out).
  • There are no "users": There is only one single user, and that is me and me alone. My MTA does not transmit to other GMail accounts.
  • I cannot follow bulk_mail.html: According to "bulk_mail.html" I must click on "No Spam" within GMail to lift this block. However I cannot click on "No Spam", as not a single(!) of those messages were tagged as SPAM. All messages are no SPAM according to the GMail SPAM filter , which is correct, as they are simply undeliverable messages.

I tried to contact the Postmasters at Google

I tried to contact the Postmasters at Google, however they did not answer. So I needed to make this Google-Bug public.

Some background information

My Domain has a catchall, so it accepts anything in front of the @ to reach me. This is because this Domain must accept Addresses of a certain form.

The From abuse now has generated random strings in front of the @. So the backlog of unreachables hits my mailbox.

That is no problem usually, as I do not send eMail with the From of that domain. So all I get is eMail directed to this domain. And the zillions of undeliverables which hit me are no problem, as they match a filter at GMail, which puts them in a trash (over 1000 per day). That's easy and convenient.

Note that in my Domain there are SPF records denying all other MTAs than mine. However SPF is of no help (Google does an SPF check, but still accepts eMail which fails SPF checks, which is extremely bad, but this is another story).

The problem now is twofold:

  • First: I cannot express the valid eMail addresses within sendmail easily, as sendmail has no way to use regexp to check an address. Well, I could hack the sendmail.cf, that's right, but this is cumbersome.
  • Second: When I just eat and discard these backlog, I help other postmasters, as these messages do no harm then. If my MTA rejects those messages, the messages are forwarded to the postmaster account at the other side.
What I now have to do is:

  • First: Tweak the MTA such, that it rejects messages which are not conforming to a certain standard.
  • Second: Apply the Floriani Principle.
What is the Floriani Principle? from en.wikipedia.org/wiki/Saint_Florian

The Austrian Floriani Principle is named after a (somewhat cynical) prayer to Saint Florian: "Oh heil'ger St. Florian
verschon' mein Haus,
zünd' and're an",
translating to
"O holy Saint Florian, spare my house, kindle others".

The Floriani Principle shows a basic wrong pattern which is evil but nevertheless very widely adopted. Most Anti-SPAM-methods applied today are based on the Floriani Principle and therefor must be considered evil as well.

The trap most people are falling in when fighting SPAM is the wrong idea of "if everybody adopts the Floriani Principle then the last one in the chain is hurt, and that are the SPAMmers". This, however, has been proven wrong already. This fact is well known for over 10 years now (since around 1994 AFAICS), but still, it is ignored. Even at Google.

Note that I, for my part, consider Anti-SPAM-methods based on the Floriani Principle are even more harmful than SPAM. Examples:

  • Some methods introduce additional messages. This way the number of messages due to SPAM is extended by some factor (factor 4 to 5). Moreover those Methods generally do not work: A sends to B. B does not see the message, instead a verify is send to A. A does not see the verify, instead a verify is send to B. Deadlock. Some people start to fight back. That is, they have robots which automatically answer verifies. This then is extremely SPAMmer friendly, as all SPAMmers have to do is to send messages with the appropriate From to get around those verification issue.
  • Some methods try to block SPAM. This often blocks legitimate eMail (recipients asked about this always arrogantly tell they simply don't care about it, which is evil). But even worse this often harms other postmasters which are innocent. Think about the fact when ORBS refused to remove IPs from the database. Well, you might think, that was the fault of the postmasters of their IPs to be listed there. However if you believe that, then you are either stupid or arrogant, but one thing is for sure: You are simply wrong. IPs are not endless. So it often is the case that you are handed out IPs which are listed on some blocking lists just because somebody who had the IP before did something wrong. And then you have the trouble to get off those zillions of lists. And this can cost you thousands(!) of dollars of working hours to do so. In nearly every case, blocking SPAM is harmful. You can see that easily if you realize that blocking SPAM is based on the Floriani Principle as well.
The Floriani Principle is no solution to a problem. The Floriani Principle only makes things worse. I do not know any exception to this.

Example:

There was a company with a rented server. The server was configured to send status eMails when in trouble. Everything was tested and it worked quite well.

Then some time later the server got in trouble. It tried to notify the operators. However the monitoring eMail did not reach them. What happened?

The eMail relay was switched from a local implementation to a service provider. This service provider had an Anti-SPAM-method in place. The Anti-SPAM-method used a blocking list. And for some unknown reason the server, which tried to send the notify, was found on this list. So the service provider rejected the eMail. The notification never went through.

The service was called to fix this problem, as the server did not work as advertised. However looking into the logs showed, that the server worked correctly, only all eMail was rejected, which caused all the trouble. Note that the service who found this was me.

You can ask yourself who paid the bill.

Conclusion: Anti-SPAM-methods based on the Floriani Principle most time introduce more problems then they solve.

Here, at my side, I have more trouble due to Anti-SPAM-methods blocking legitimate eMail than I ever had due to SPAM. Well done!

-Tino, 2008-04-23