[Greylist-users] Some more data points

[Scott Nelson]

At 11:50 AM 7/1/03 -0500, Evan Harris wrote:
>
>On Tue, 1 Jul 2003, Scott Nelson wrote:
>
>> Results of my testing so far;
>> Out of 1102 attempts, 515 succeeded.  Roughly 50%
>> Most of those successes seemed to come near the end of my trial.
>
>Over what period?  
>
9 days.

>How many of those attempts were unique triplets?  
>How many of the successes were unique?  

1102 unique triplets.
515 successful deliveries.
203 "unique" deliveries.
312 "repeat" deliveries.

>How many destination accounts were there?  
>
100, 10 @ 10 different domains, but all in the same netblock /28.


>Is this using greylisting by itself, or were you also using
>spamassassin and/or rbl lists?  
>
Greylisting only - I whitelist my own address space, 
but there wasn't any email from it to the spam traps.

>Does the succeeded number you quote count only actual passed emails, 
>or is this before other mail checks, like invalid
>recipients?
>

These are only emails destined for the 100 spam trap accounts
that have greylisting turned on.

I don't count spam delivered to non-greylisted accounts,
nor spam blocked because the account is non-existent.

How can you know if a non existent account wanted greylisting
turned on?


>> Either I've goofed up somehow, or some spammers have already
>> adapted to greylisting.
>> Has anyone else noticed a sudden increase?
>
>You probably got spammed by one spammer at the end who was using some type
>of real mailer that retried, and threw off your numbers.  A thousand
>attempts over about a week (I'm assuming the timeframe) isn't a very large
>sample size.
>

1000 isn't exactly huge, but +/- more than twice the square root
of the sample size is unlikely.  
A variance larger than 10% should be extremely unlikely.
My numbers for the first 6 days are /dramatically/ better - over 90%.
That doesn't add up, and since no one else mentioned a similar increase, 
I'd have to go with "I must of goofed up"

Counting the messages in the spam mailboxes shows only 
290 messages waiting, and some of those are from today.

Sigh... 

Ok, I'm going to declare this run a bust, 
and start over as soon as I figure out what I did wrong.


>> A couple of notes;
>>
>> In other tests I ran, there was a marked difference in successes
>> rates when tempfailing after the RCPT rather than after DATA.
>> Eyeballing my logs, I notice a lot of instant retries on a different
>> IPs after failure, usually three times.
>
>That's why I tend to favor reporting by unique triplets.  That removes these
>types of accounting errors, even though I didn't see that many of them.
>

Uh, no.

Here's a snippet from my logs;

Recvmail[28439]: connection from 200.86.197.182 at Tue, 1 Jul 2003 08:07:22 -0700
28439: s: 220 slyrat.com ESMTP Ready
28439: c: Helo dnKA58
28439: s: 250 slyrat.com Hello dnKA58, pleased to meet you
28439: c: MAIL FROM: <maryann at nest-box.com>
28439: s: 250  <maryann at nest-box.com>... Sender ok
28439: c: RCPT TO: <bob3 at slyrat.com>
28439: triplet-231edfcd added
28439: s: 451 Please try again later
28439: c:
28439: bye bye child -1
21869: Reaping 28439

recvmail[28440]: connection from 200.44.112.35 at Tue, 1 Jul 2003 08:07:31 -0700
28440: s: 220 slyrat.com ESMTP Ready
28440: c: Helo L8Jvjq
28440: s: 250 slyrat.com Hello L8Jvjq, pleased to meet you
28440: c: MAIL FROM: <maryann at nest-box.com>
28440: s: 250  <maryann at nest-box.com>... Sender ok
28440: c: RCPT TO: <bob3 at slyrat.com>
28440: triplet-d76d9741 added
28440: s: 451 Please try again later
28440: c:
28440: bye bye child -1
21869: Reaping 28440

recvmail[28441]: connection from 212.22.48.185 at Tue, 1 Jul 2003 08:07:36 -0700
28441: s: 220 slyrat.com ESMTP Ready
28441: c: Helo 1m5UCw
28441: s: 250 slyrat.com Hello 1m5UCw, pleased to meet you
28441: c: MAIL FROM: <maryann at nest-box.com>
28441: s: 250  <maryann at nest-box.com>... Sender ok
28441: c: RCPT TO: <bob3 at slyrat.com>
28441: triplet-c83a7aae added
28441: s: 451 Please try again later
28441: c:
28441: bye bye child -1
21869: Reaping 28441


Note that the envelope from is the same, the envelope to is the same,
only the IP changes.  And note how quickly they switched from one
to the other.  There are several similar attempts in my logs,

If the delivery had been accepted, 
I assume it would have only happened once.
As it stands, it's three unique triplets, three attempts, all blocked.

That's why I think comparing absolute spam number against a control
group is necessary - there's just no way to know how much spam
would have been delivered unless you actually accept it.


>Keep in mind that if you have several MX servers for your domain(s) and are
>using the same db for all, you'll often see an increment to the blocked
>count for each MX host, since many legit servers (and spammers) will try all
>MX hosts for a domain one after another when they recieve a tempfail from
>one.
>

I don't have multiple MXes, but I'll keep it in mind.


>> It occurs to me that an unscrupulous anti-spam company could improve
>> their spam catching /percentages/ by spamming themselves,
>
>That's entirely true.  Or, they could "seed" their domain with fake
>addresses on the web for spammers to harvest, and count those toward their
>totals too.
>

Well they could lie outright about the numbers too,
but if you stuff the quarantine folder with your own spam
it's a little different.  (Ok, maybe it's not that different,
but it feels different)


>> If I do any future testing, I plan to compare results against
>> a control group.  Comparing the total number of spam actually received
>> at addresses that have whatever anti-spam technique, to spam received
>> at addresses that do not.  It's more work, but I think it's necessary.
>
>That's an excellent idea.  Unfortunately unless you have a very large test
>site and a large control site, your comparisons will probably have a large
>error, since spamming by it's nature has pretty widely varying fluctuations,
>since you can't be sure that both sites are on all the same spammer lists.
>

Yep, that's the problem alright.

Scott Nelson <scott at spamwolf.com>