From craig@hughes-family.org  Mon Sep  2 13:12:49 2002
Return-Path: <craig@hughes-family.org>
Delivered-To: yyyy@localhost.netnoteinc.com
Received: from localhost (localhost [127.0.0.1])
	by phobos.labs.netnoteinc.com (Postfix) with ESMTP id 7600944161
	for <jm@localhost>; Mon,  2 Sep 2002 07:38:06 -0400 (EDT)
Received: from phobos [127.0.0.1]
	by localhost with IMAP (fetchmail-5.9.0)
	for jm@localhost (single-drop); Mon, 02 Sep 2002 12:38:06 +0100 (IST)
Received: from blount.mail.mindspring.net (blount.mail.mindspring.net
    [207.69.200.226]) by dogma.slashnull.org (8.11.6/8.11.6) with ESMTP id
    g81B83Z21386 for <jm@jmason.org>; Sun, 1 Sep 2002 12:08:03 +0100
Received: from user-1120fqe.dsl.mindspring.com ([66.32.63.78]
    helo=belphegore.hughes-family.org) by blount.mail.mindspring.net with
    esmtp (Exim 3.33 #1) id 17lSaV-0007tD-00; Sun, 01 Sep 2002 07:08:16 -0400
Received: from belphegore.hughes-family.org (belphegore.hughes-family.org
    [10.0.240.200]) by belphegore.hughes-family.org (Postfix) with ESMTP id
    BEB2D3C545; Sun,  1 Sep 2002 04:08:14 -0700 (PDT)
Date: Sun, 1 Sep 2002 04:08:14 -0700 (PDT)
From: Craig R Hughes <craig@hughes-family.org>
Reply-To: craig@stanfordalumni.org
To: Daniel Quinlan <quinlan@pathname.com>
Cc: "Craig R.Hughes" <craig@deersoft.com>,
	Justin Mason <jm@jmason.org>,
	SpamAssassin Developers <spamassassin-devel@lists.sourceforge.net>
Subject: Re: [SAdev] results of scorer evaluation
In-Reply-To: <yf27ki6cco2.fsf@proton.pathname.com>
Message-Id: <Pine.LNX.4.44.0209010357090.5726-100000@belphegore.hughes-family.org>
MIME-Version: 1.0
Content-Type: TEXT/PLAIN; charset=US-ASCII

Daniel Quinlan wrote:

DQ> Before we release, it'd be great if someone could test a few
DQ> additional score ranges.  Maybe we can lower FPs a bit more.  :-)

I don't think there's much more room for lowering FPs left which the GA can 
achieve.  Remember, also, that the AWL will reduce FPs, but its effects aren't 
factored in to the GA scores.

The work currently being done on the GA, and comparing different methods of 
doing the score-setting, is very worthwhile, and extremely useful; however, we 
really ought to get a release out, since 2.31 is getting decreasingly useful as 
time goes on.

The FP/FN rate of 2.40 with pretty well *any* score-setting mechanism will be 
better than 2.31 -- we can continue with adjusting how the scores are set on the 
2.41 or 2.50 branches.

DQ> Something like:
DQ> 
DQ>   for (low = -12; low <= -4; low += 2)
DQ>     for (high = 2; high <= 6; high += 2)
DQ>       evolve

You could just allow low and high to be evolved by the GA (within ranges); I'd 
be enormously surprised if it didn't end up with low=-12 and high=+6, since 
that'd give the GA the broadest lattitude in setting individual scores.  The 
issue with fixing low and high is not one of optimization, but rather one of 
human-based concern that individual scores larger than about +4 are dangerous 
and liable to generate FPs, and individual scores less than -8 are dangerous and 
liable to be forged to generate FNs.

DQ> Maybe even add a nybias loop.

Adding an nybias loop is not worthwhile -- changing nybias scores will just 
alter the evaluation function's idea of what the FP:FN ratio should be.

DQ> > AFAIK there's nothing major hanging out waiting to be checked in 
DQ> > on b2_4_0 is there?
DQ> 
DQ> Nope.

Great!

DQ> > I'll be on IM most of today, tomorrow, and monday while cranking 
DQ> > on the next Deersoft product release (should be a fun one).  Hit 
DQ> > me at:
DQ> > 
DQ> > AIM: hugh3scr
DQ> > ICQ: 1130120
DQ> > MSN: craig@stanfordalumni.org
DQ> > YIM: hughescr
DQ> 
DQ> We've been hanging out on IRC at irc.rhizomatic.net on #spamassassin
DQ> (the timezone difference gets in the way, though).

I've been searching for that, but I guess the details of where the channel was 
got lost in the shuffle.

C


