TODO

The ideas presented here have been discussed on the mj2-dev mailing
list.  Items should get added here when it is decided that something
needs to be done, and removed when the folks who asked for the
feature/fix have agreed that the feature is in/problem is fixed, or that
the feature is not needed/problem really isn't a problem.

==================================
Implemented, awaiting verification
==================================

Better Queue Handling
---------------------
If a message causes Majordomo to crash, it remains in the queue and will
continue to cause crashes.  Thus it is a good idea to rename queue files so
they don't get rerun.  However, in the case of system crashes/random
Majordomo crashes/other nonrepeatable issues, you _do_ want the queue file
processed.  This it is advantageous to have a processing count and to only
process files some number of times normally before giving up on the file.

Also, giving up on a file is not good, as it can lead to problems not being
noticed.  Thus the idea procedure would be to process a queue file normally
twice (or thrice) and then on the next try, to turn up debugging and log to
a special file.  The next try, delete the queue file.  The owner also needs
to be informed somehow.

To manitain the generation count (and perhaps additional data we might want
to keep) mj_queuerun will maintain a status file named by prepending the
message filename with a '.'.  In it is stored a single line with a single
number: the attempt count.  No locking of the status file is necessary.

This is implemented as requested, but still needs tweaking.  The queue file
should be saved off somewhere.


===========================
Not yet (fully) implemented
===========================


Configuration Rewrite
---------------------

The current configuration process is incompatible with modern package
managers.  Defaults cannot be hardcoded into the scripts.  Only two things
can be: the path to Perl and the location of the config file.  Everything
else should be put into a separate configuration file.  This cound (should)
subsume the SITE config file currently used for defaults that aren't
hardcoded.

The config file should probably be raw Perl code so that it is as fast as
possible to load.  The Q&A section of Makefile.PL can be extracted and
turned into an "easy setup" script.

Also, the temporary directory (and probably several other variables) needs
to be made part of the site config instead of a per-domain configurable.

Things currently hardcoded in the scripts:
  $::BINDIR   - OK to stay
  $::LIBDIR   - OK to stay
  $::LISTDIR  - move
  $::UMASK    - move
  $::TMPDIR   - move
  $::LOCKDIR  - move
  $::WTMPDIR  - move
  $::UID      - both (need for install, but put in config file)
  $::GID      - both
  $::TIMEOUT  - move
  $::CONCURRENCY - move
  $::SIG_CHLD_IGNORE - OK
  $::LOCK_EX   - OK
  $::LOCK_NB   - OK 
  $::LOCK_UN   - OK
  $::O_WRONLY  - OK
  $::O_CREAT   - OK
  $::O_EXCL    - OK

Things currently in SITE/config.pl (all should move):
  cgi_bin
  install_dir
  mta_separator
  queue_mode
  database_backend
  mta
  mta_options (MTA specific)
  site_password
  cgi_url


Wrapper Security
----------------
The wrapper needs to sanitize as much of its environment as possible.  This
involves making a list of needed env variables and deleting everything else.


Uuencode and ms-tnef in the body as "attachments"
-------------------------------------------------
Treat uuencode and ms-tnef chunks in the body as attachments so that
attachment_rules can be used to trap or remove them.

Come up with some fake MIME type used to represent these "attachments" so
that they can be put in attachment_rules.  "uuencode" and "ms-tnef" are
probably sufficient, although there is potential for confusion because a
rule of "uuencode | discard" wouldn't discard a real MIME part of type
"application/x-uuencode".

When checking for taboo lines, if the above fake mime types are referenced
in attachment_rules, scan the body for a sequence which identifies them and
trap the message accordingly.  (How to check if they're referenced?  Run
the rules code and check the result.)

When modifying the body, if the above fake mime types are marked as
"discard" in attachment_rules, scan the body for the identifying sequence
and drop the whole body part up to and including the ending sequence.
Insert a message indicating that the data was deleted.


MTA configuration hook
----------------------
Many MTAs need special programs to be called to build database files out of
the aliases and associated files which we generate.  This is not generally
possible to do within Majordomo for two reasons: 1) it is site-dependent
and 2) it requires priviliges that Majordomo doesn't have (frequently
root).

To solve this add a "hook": look for a program in a specified place and, if
it exists, call it with some defined calling convention.

For many sendmail installations, it suffices to call newaliases, although
on newer sendmails this must be done as root.  Other installations need
calls to makemap as well.

Possibilities: 1) setuid program to exec newaliases 2) touch a file, which
is checked by a root-owned cron job that runs newaliases and makemap if
it's present.

Exim can use this feature to use databases instead of flat files, which
would speed up installations with large numbers of domains or lists.

Suggested script location: $install_dir/libexec/build_databases.  Not in
  the normal bin directory, because this is not a user executable.

libexec chould be mode 700, owner majordomo, group majordomo.  (This is
  security sensitive, and should be restricted beyond normal majordomo
  files.)

Arguments taken: none; the script gets called any time a change is made and
  should generate all databases.

Permissions: 700, owned by majordomo or 750, owned by root, in majordomo
  group.


mj_queuerun lifetime limiting
-----------------------------
In addition to an idle timeout, mj_queuerun needs a limit on how many
messages it will process without exiting.  This keeps memory leaks under
control.  The queue server will simply start another runner when one exits,
so this is a safe cleanup mechanism.  (Currently, each runner has a
hard-coded limit of 1000 messages.)


mj_queueserv bug with all dead queue runners
--------------------------------------------
mj_queueserv has a bug where it will queue the message due to excessive
load when in reality every single queue runner has died.  This happens
because it runs through all of the slots looking for a live and unbusy
runner, cleaning up dead ones along the way.  But it needs to restart the
search if it's cleaned up any dead ones but has run out of slots without
finding any waiting runners.


Making use of FastCGI or something like it
------------------------------------------
Startup time is painful, and startup time of the web interfaces is
especially painful since every button click will open another URL and
compile another script.

There are tools which maintain pools of service threads to handle incoming
requests; FastCGI is just one.  Some don't work with setuid programs.  I'm
pretty sure that one of them will work for us.


Stale queue entries
-------------------
It is rare but possible for a message to sit in the incoming queue with no
active queue runners; the message will sit around until another message
comes in and kicks off a queue runner, which will then take care of both
messages.  There should be a way to force a queue run without sending a
message.


Queue view
----------
There should be a way to view the message queue.  A separate program,
mj_mailq, would be sufficient. 


Load balancing
--------------
Configure a list when it is created to use digest and triggers settings
randomly chosen from a collection of possibilities.  This would
distribute the periodic load due to digest deliveries more evenly.


Auxiliary lists
---------------
Convert the owners and bounce_recipients addresses into auxiliary
lists to support address validation and access controls.
(DEFAULT values should continue to be available.)

Note that this has implications for how we choose moderator groups and
such.  If we assume that the list is small, then we can just pull out the
addresses.  But some features of sublists would then not be available.


Access variables for moderators/owners
--------------------------------------
Barring the previous item, we should at least have $is_owner and
$is_moderator access variables.


BerkeleyDB database backend
---------------------------
Instead of DB_File, use the BerkeleyDB module to gain access to the
interesting features of BerkeleyDB (CDB, mainly, to avoid some very
expensive locking crud).

This should be pretty easy to get started, but the more esoteric features
will be interesting.


Filesystem database backend
---------------------------
The filesystem _is_ a database.  Why not use it?  A database is a
directory, a key is a filename and its value is its contents.  It may be
possible to do away with lots of nasty locking (or to use record locking,
or somesuch).  Space waste can be huge for large blocksizes on filesystems
that don't do tail packing.  Reiserfs on Linux is optimized for precisely
this kind of thing.


Type I envelopes
----------------
We need an envelope type 'I'; any bounce received with this type is
completely ignored.  General responses from majordomo should be sent out
with this type.


bounce_rules for GLOBAL 
-----------------------
If a message bounces to majordomo-owner, the domain owner sees it but
cannot turn it off.  It should be very easy to make bounce_rules apply to
GLOBAL.  Tracking of bounces may need to be turned off for GLOBAL, though.


Automatic backup and recovery of subscriber databases
-----------------------------------------------------
The who-export command could be executed periodically to save the
subscriber list and settings for a mailing list.  This could also be
done incrementally.


Display subscription statistics
-------------------------------
For accounting purposes, it would be useful to display statistics about
the subscribers of a mailing list.  The statistics would show the number
of subscribers in each delivery class, and how those numbers have
changed over time.


Virtual User Table support for Postfix
--------------------------------------
The current virtual user table support assumes that the MTA is Sendmail.  


Better tmpdir and debug support
-------------------------------
The default value during installation should be on a file system that
can hold hundreds of megabytes.  It should be possible to rotate logs
and expire old files automatically.


Simplify the wwwadm list configuration
--------------------------------------
Remove the requirement to check a box of each setting to be changed.
Complete the assignment of settings to wizard levels.


Display a summary of list categories
------------------------------------
To make the lists command more scalable, implement a lists-categories
command to see a summary of categories and a lists-category command
to display lists whose category matches a pattern.


Hide addresses in mj_wwwusr archives
------------------------------------
To protect the archives from address harvesters, add substitutions
for the local part and domain name of the author of each message.
Possibly, attempt to identify or obscure domain names in the bodies
of messages.


Make past digests retrievable
-----------------------------
Currently, no record is kept of which messages were contained in a 
particular digest volume/issue number. 


Add cookie support to WWW interfaces
------------------------------------
Links to external sites can cause latchkeys and other information to
be leaked in HTTP headers.  Adding cookie support would alleviate
this problem.  It would make the continuation of sessions more
convenient.


Remove hard-coded English
-------------------------
A variety of error messages, moderation reasons, and results from the
mail parser are still hard-coded in English.  For example, the
documents stored by the newfaq, newinfo, and newintro commands
use hard-coded English descriptions.


Improve names of commands and settings
--------------------------------------
The bounce_max_age setting should be renamed to bounce_lifetime for
consistency.


Support hours and minutes in reports
------------------------------------
To support hourly reports, extend the time syntax.
(e.g., "200309010700" would mean "7 am, 1 September 2003").
Also, allow compound time specs in the "report" command syntax.


Improve domain adjustments
--------------------------
The "make domain" command should allow a domain to be removed or
renamed.


Web posting
-----------
The wwwadm and wwwusr interfaces cannot currently be used to post
a message or reply to an archived message.


=======================
Totally out there stuff
=======================

There are some things that I (Jason) have always wanted to experiment
with.  One day I will.  Here are some of my ideas.


server-side killfiles
---------------------
Allow users to make simple killfiles, like "subject contains blah" or
"author is annoying@aol.com".  Very limited functionality: no pattern
matching, don't match headers.

Applying them efficiently is actually pretty easy.  When adding a new list,
assign it a number, store a big list of them somewhere, and store only the
number in the subscriber data.  Apply the whole lot of them at once.  Keep
track of the ones that matched in a hash, then during delivery check to see
if the numbers in the subscriber data are in the hash.  If so, don't
deliver to them.


issue tracking
--------------
Issue tracking and mailing lists are closely intertwined.  Any real
issue tracker should read the list and keep track of discussion relating to
open issues.

issue tracking can be very simple.  Issues can be open or closed.  They may
have reporters, owners, dispositions, categories, etc..  Basic operations
on issues are: open, close, take, give and set some property.

The cool part comes when you manipulate the issue tracker as part of the
list discussion tracker, by including commands in the message body.  Say
someone reports a bug.  You reply to it with your comments and stick "open
issue" or something at the top of the body.  The list server sees this and
sticks "[issue #23]" in the subject before the message is resent.  Replies
that have this subject get archived with the issue.  When the issue is
resolved, the message stating that (which could even be a CVS commit
message) just needs to include "close issue 23" at the top of the body.

Issues can be queried and manipulated via the normal command interface.
The server can generate reports of open issues.


polling/voting
--------------
There's a voting package which integrates with Mj1.  It would be
interesting to see if something that works with Mj2 can't be cooked up.


automatic taboo generation
--------------------------
I've seen mention of a system that automatically generates taboo roles for
Mj1 based on spam received at various spamtrap addresses.  We should try to
hook into this if possible.


automatic newsgroup gatewaying
------------------------------
Folks have asked for injection of list messages into newsgroups.  There's a
Perl module for this, so it shouldn't be a huge issue.  Bidirectional
gatewaying is tougher, but it should be a solved problem.


amavis-perl integration
-----------------------
There's a nice virus scanner writtein in Perl called amavis.  We should
look at interfacing with it.  We might be able to lift some ideas from it
as well, since it has a daemon mode.