evan_tech -- removing duplicates using formail

09:33 am, 26 Nov 04

removing duplicates using formail

An offlineimap hiccup from a long time ago put two copies of every message in my inbox. formail has a duplicate filter: it stores every message id in a cache file and when it sees a messageid twice, it returns success. If you add in the -s (split mail) flag, it will only output any message once.
So:
formail -D 10000000 cache -s cat < mbox > mbox2

Then to convince offlineimap to not recopy all of your local duplicates of messages back up to the server, rm your local copy of this mailbox and all mentions of it in ~/.offlineimap and its subdirectories. The next run will cause a full download of that mailbox.

(No work today, so it's mail-cleaning time: adding more data to the spamassassin bayes learner, removing duplicates, and next I'll try that bounce-filtering scheme y'all suggested.)

zanfur

Bounce-filtering? Are you referring to temporarily rejecting it if it seems like it might be spam, with the idea that if it's legit it will get resent? That scheme works pretty well, but the mail often doesn't get re-sent until half an hour later, which can get really frustrating with coworkers who have html footers. I like the greylisting options better (uses moving averages of spam scores to determine spam/ham), but with the volume of mail going through my server it just can't handle the load.

I never did get bayes working. I think I'll go read up on that again.

reply to this

evan

briefly:
i get a few "returned mail: user unknown" messages a day, from stupid mail servers that think the spammers "forging" my from: are actually mails from me.
the tactic to fixing it is to add an extra header to all outgoing mail, and then procmail away all bounce messages that don't contain the header. (bounce messages contain all the headers of the original message, y'know?)

ugh, i hate mail.

reply to this

zanfur

That makes sense. But, can't you just look for your server's hostname/ip in the "Received-From" headers instead of adding a nonstandard header that might get stripped out? Some stupid mail exchangers (*cough*exchange*cough*) strip out headers they don't understand. I can't think of any mail exchangers that strip out received-from headers, except spam relays specifically configured that way.

You must get scads of spam. Do you rely solely on the bayes filter, or do you have static rules as well? Any non-standard rulesets? I'm really interested in how you've got your mail server set up.

reply to this