[Greylist-users] greylist lib in C? + several Q's

[Ken Raeburn]

"William Blunn" <bill--greylist at blunn.org> replies, to Graham Toal:
>> Do you decide
>> to either pass the mail or not, for all users, or do you
>> reject all recipients except one
>
> Again, if you do it at RCPT time, you can return results for each
> recipient.
>
> If you do it somewhere else, you need some way of combining the
> greylisting results for all the recipients.  This is an implementation
> decision.
>
> The algorithm I used was to take the "most rejectional" result out of
> all the results for all of the recipients.  The possible greylisting
> results for each recipient are:
>
>   0 ACCEPT
>   1 TEMPREJECT
>   2 REJECT
>
> Then take the maximum value reached for this messsage, and that is the
> result for the message.

So if the email is to be (permanently) rejected for one of the
recipients, and accepted for others, you'd issue a reject after the
DATA phase?  I hope you don't allow random users to blacklist senders,
then.


>> and cause the sender to
>> send individual copies to each person so you can make the
>> decision on a per user basis?
>
> I am unable to relate your question to my understanding of e-mail
> delivery.  Re-tries are done by the sender's MTA.  The sender's MTA will
> track the delivery status for each recipient, so if you accept some and
> tempreject others, the sending MTA will re-try only the ones which were
> temprejected.

... and won't do so individually.

>> 2) Is there any danger in *always* doing the temporary
>> reject after the DATA command is complete?  I know that the
>> whitepaper suggests doing this only for MAIL FROM:<>
>> (with some hacks for broken mailers) but for my purposes
>> I'd rather like to do it that way all the time.  One
>> reason being I want to store the mail, for QA purposes,
>> so we can be sure that good mail has not been rejected;
>
> You're not rejecting it, you're temporarily rejecting it.
>
> Your MTA is *always* entitled to tempreject.  The load average could be
> too high.  That is what temporary rejection is for.  If the sending MTA
> doesn't re-try, then that is *their* problem, *you* are golden, *they*
> don't have a leg to stand on.

Try explaining that to the people at Yahoo Groups.  Try explaining to
your users that Yahoo Groups misbehaves, and therefore isn't allowed
to send them email.

No, small sites you might be able to get fixed or ignore, but unless
you've got a lot of leverage, there will be the occasional broken
large site that you can't fix and can't afford to lose lots of mail
from.  For the near future, you'll need a whitelist, and it will need
to be maintained manually, and ideally you'd want to be able to
monitor traffic for possible updates to the whitelist.  If you don't
mind losing an occasional message, maybe you can do it with logged IP
addresses and hostnames; if you're more paranoid than that, you want
to collect message bodies for at least automated analysis.
Unfortunately that's not very compatible with tempfailing for some
recipients and not others.

I'm somewhat on the paranoid side, but even I'm happy with Evan's
relaydelay and the risk of the occasional lost message from
misbehaving sending sites.

>> another is that I'm considering greylisting *only* if it
>> fails a spam test - otherwise it is accepted.
>
> I'm not sure that this is a good idea.  Greylisting tends to be the
> better initial triage function as it can weed out a vast proportion of
> incoming delivery attempts at very low cost.

Yep.  Most reports are that it cuts down CPU and network resource
consumption by causing you not to ever look at a large fraction of the
(supposed) spam traffic.

Another approach I've deployed recently that also seems to help is
delaying the greeting banner.  The spec says the client has to wait
for the banner (for some reasonably long time like 5 minutes) before
it sends anything.  Spamware and viruses often do not.  So, wait 45
seconds or so before sending the banner, and if any traffic comes in,
it's not a properly behaving MTA, so reject everything it sends, or
drop the connection.  If they disconnect on you, they're also not
following the specs.

With the 45s delay, my mail system at home has been discarding about
half as many attempts as greylisting later tempfails (more, in the
last day or so).  Blithely assuming that all of those would result in
database entries, that could be somewhere around 1/3 of the potential
spammer entries in the greylist database no longer getting added.

I chose 45 seconds based on a web page I found (and can't find again
at the moment) where someone had done some monitoring on a system set
to delay 90 seconds, and found that something like 90% of hosts that
misbehaved did so in the first 45 seconds.  I suppose I could go for
90 or 120 seconds and get a few more....

(Sendmail 8.13 has this capability built in; it's about 60 lines of C
code that can easily be dropped into earlier versions, and a little
addition in sendmail.cf.)

>> This ought
>> to cut down the risk of delays to legitimate mail, which
>> appears to be a concern here.
>
> In my experience greylisting generates very few false positives.
>
> Are you a business?  If so, then it is a cost/benefit analsys.
>
> What is the cost of a very few number of false positives which you would
> have with greylisting compared to the cost of the enormous amount of
> spam and virus messages which greylisting would stop?

Perhaps doing the tempfail-after-data approach would be a useful way
to do the cost/benefit analysis.  Say, spend a couple months doing it
that way and seeing which messages really get retried and which are
spam, but actually delivering all messages the first time (suppressing
duplicates by recording sender, recipient and sha-1 checksum, maybe).
Then no mail has been lost, and you've got a couple months' worth of
data to analyze.

Be careful of assuming that some automated spam filter always gives
the correct result, however.  If greylisting (or any other technique)
would've stopped message X from being delivered, and spamassassin says
X isn't spam, does that mean that greylisting was unreliable, or that
the spammer tuned his message to get around spamassassin?  But privacy
policies may mean you can't evaluate for yourself whether a saved
message is really spam, if it's not for you.

You could also use the saved data to look for sites that use multiple
sending hosts, or that don't retry even though a majority of their
messages are not spam.  You can build up your whitelist from those
sites.  Sites that send you a lot of traffic and always queue and
retry might also belong on your whitelist.

>> If we do store the mail
>> for a certain time, waiting to see if it is resent, would
>> a simple hash function allow me to recognise the same
>> mail the next time round or does mail change in small
>> ways when it is resent?  (We need to recognise resent
>> mail in order to take it out the store, so that any
>> remaining mail after the retry expiry delay must be
>> the spams that we rejected)

The message coming across the wire should be the same both times, for
any MTA I know about.  But note that the local MTA does add a
"Received:" header, so if your hash function is applied after that
header is added, it should be excluded, along with any other
locally-added timestamp or indication of sending IP address (in case
you're dealing with an ISP with a pool of mail-sending systems).

>> 4) What is the longest observed delay between first attempt
>> and the retry, for a legitimate sender?
> Some MTAs re-try based on the target HOST, so if you have a new
> greylisting set-up and you get a fair number of messages coming through,
> say, an ISP mail relay, then you can get some nasty pathological cases
> in the beginning. These usually settle down after a few weeks.

If you mean the ISP mail relay is acting as an MX site for you, that
relay should be whitelisted.  Even if it's outgoing mail from another
ISP that queues and retries, if you get a lot of traffic through
there, you might as well whitelist them, unless there's a reason that
delaying messages you know will still get delivered to you might help
(e.g., comparing messages against a spam message database that gets
updated by other people in real time).

>> 8) The whitepaper suggests storing the arrival and expiry times.
>> Is there a reason for storing anything other than the arrival
>> time?

By "arrival" time I assume you mean the time the triplet is first
seen?  I don't think there's a great need to store that, unless you
want to see how long you've been getting mail regularly (i.e., at
least once every 36 days) from a particular source.

The expiry time you need, either stored or computed.  In Evan's
implementation, the expiry time is also used to implement a whitelist
or blacklist entry -- have the record expire in the distant future
(date 9999-12-31), and have the block expire either in the past
(whitelist, year 0000) or in the distant future (blacklist, now plus
one year recommended).  Personally, I ignore the blacklist
recommendations and only use the whitelist.

> (In fact the Bagley system, unlike every other implementation I have
> seen, works entirely in UTC, thereby eliminating the problems which
> would otherwise occur at the daylight-savings/non-daylight-savings
> changeover.)

Interesting point, I hadn't thought about that.  Perhaps Evan's
implementation should be updated to do that too.

> The Bagley system also stores a status value against each triple, so
> that if, for example, the greylisting delay is extended, you do not get
> triples which were previously happy suddenly start temprejecting again.

So you've kind of got a block-expired flag instead of a block-expires
time...

>> The expiry time is calculated by a simple addition of a
>> constant, but if you change your policy, wouldn't you want it
>> to apply retroactively to all entries in your database rather than
>> just new ones being added?

Perhaps.  I think it would be an infrequent event, and the problem
automatically handled in time as older records expire or get updated.
If you really want to make it retroactive, I think running a simple
SQL command to add the delta in the appropriate cases wouldn't be to
much to ask.

Ken