Persistent class.

Add -w and -W option to dump the word list (by word and by wid,
respectively).

Except KeyboardInterrupt from unqualified except clauses.

- Use slightly more portable values for the Data.fs and
  Zope/lib/python.

- Add -t NNN option to specify how often to commit a transaction;
  default 20,000.

- Change -p into -p NNN to specify how often (counted in commits) to
  pack (default 0 -- never pack).

- Reworked the commit and pack logic to maintain the various counters
  across folders.

- Store relative paths (e.g. "inbox/1").

- Store the mtime of indexed messages in doctimes[docid].

- Store the mtime of indexed folders in watchfolders[folder] (unused).

- Refactor updatefolder() to:

  (a) Avoid indexing messages it's already indexed and whose mtime
      hasn't changed.  (This probably needs an override just in case.)

  (b) Unindex messages that no longer exist in the folder.

- Include the folder name and the message header fields from, to, cc,
  bcc, and subject in the text to be indexed.

valid value is input, or the empty string, and interpret the empty
string as the default.  Indicate the default in the prompt.

Add glob support to the HTMLWordSplitter class.

selected in a mutally exclusive manner (such as splitters).

Existing pipeline elements have been grouped appropriately.

Added a stop word remover that does not remove single char words.

Modified ZMI lexicon add form to use pipeline element groups to render form.
Groups with multiple elements are rendered as selects, singletons are rendered
as checkboxes.

but the pattern may not begin with a glob character (else
someone specifying "*" as the pattern can tie up the CPU for
a long time).

class

and recognizes the header attribute

  * A pipeline factory registry now allows registration of possible
    pipeline elements for use by Zope lexicons.

  * ZMI constructor form for lexicon uses pipeline registry to generate form
    fields

  * ZMI constructor form for ZCTextindex allows you to choose between
    Okapi and Cosine relevance algorithms

instead of an extension type, and let StopWordRemover be a Python class
that uses the helper if available.

already ditched Python 1.5.2.  The version of tempfile is many
revision behind the one in the Python std library.

Remove redundant import.
Ensure that ZCTextIndex implements the PluggableIndexInterface by
adding an unimplemented uniqueValues() method.

(similiar to getPhysicalPath())

Verify that ZCTextIndex implements the PluggableIndexInterface.

neither 'pass' (v 1.2) nor 'break' (v 1.3) but 'continue'.

Whitespace normalization.

loop-invariant, save a little time by multiplying idf by 1024. outside
the loop.

the number of words in the index (at least to return a number
comparable to the number displayed under "# objects" by TextIndex).

Index management screen.  Ditto for clear().  So group them together
and adjust the comment.  (So is manage_main, but since it's a DTML
method, it can stay in its separate UI group.)

still only supports a trailing *, so the pipeline should honor that;
added a comment to the Splitter class referring to globToWordIds().

well check it in.  This yields an overall 133% speedup on a "hot" search
for 'python' in my python-dev archive (a word that appears in all but
2 documents).  For those who read the email, turned out it was a
significant speedup to iterate over an IIBTree's items rather than to
materialize the items into an explicit list first.

This is now within 20% of simply doing "IIBucket(the_IIBTree)" (i.e.,
no arithmetic at all), so there's no significant possibility remaining
for speeding the inner score loop.

creating it anonymously and then pulling it out of the zc_index
object.

once we have more than one on the menu.)

in percentages; strip the percent sign to avoid a traceback calling
int() when these variables are used.