David Mertz, Ph.D.
Professional Neophyte
October, 2005
Welcome to "Mail and News", the second of seven tutorials covering intermediate network administration on Linux. In this tutorial, we discuss how to use Linux as a mail server and a news server. Email is probably the main use of the Internet, overall; and Linux perhaps the best platform for running email services on. This tutorial addresses mail transport, local mail filtering , and mailing list maintenance software. Server software for the NNTP protocol is discussed briefly.
The Linux Professional Institute (LPI) certifies Linux system administrators at junior and intermediate levels. There are two exams at each certification level. This series of seven tutorials helps you prepare for the second of the two LPI intermediate level system administrator exams--LPI exam 202. A companion series of tutorials is available for the other intermediate level exam--LPI exam 201. Both exam 201 and exam 202 are required for intermediate level certification. Intermediate level certification is also known as certification level 2.
Each exam covers several or topics and each topic has a weight. The weight indicate the relative importance of each topic. Very roughly, expect more questions on the exam for topics with higher weight. The topics and their weights for LPI exam 202 are:
Topic 205: Network Configuration (8) Topic 206: Mail and News (9) Topic 207: Domain Name System (DNS) (8) Topic 208: Web Services (6) Topic 210: Network Client Management (6) Topic 212: System Security (10) * Topic 214: Network Troubleshooting (1)
Welcome to "Mail and News", the second of seven tutorials covering intermediate network administration on Linux. In this tutorial, we discuss how to use Linux as a mail server and a news server. Email is probably the main use of the Internet, overall; and Linux perhaps the best platform for running email services on. This tutorial addresses mail transport, local mail filtering , and mailing list maintenance software. Server software for the NNTP protocol is discussed briefly.
To get the most from this tutorial, you should already have a basic knowledge of Linux and a working Linux system on which you can practice the commands covered in this tutorial.
The breadth of use of Linux for mail and news servers has led to the
development of improved tools over time. When the LPI certification
exams were developed, the most popular tools were sendmail for mail
transport, procmail for local mail handling, majordomo for mailing
lists, and innd (InterNetNews daemon) for NNTP. The last of these is
still probably the default choice; however, despite its technical
strengths, the NNTP protocol has been somewhat eclipsed by either
email mailing list or web-based discussion forums.
Of other tools, sendmail and procmail are still widely used,
although not as ubiquitous as they might have been in 2001. However,
the most popular upgrade/replacement for sendmail is postfix which
contains facilities for backwards compatibility with sendmail. The
field for local mail handling is well populated, but procmail is
largely used. On the other hand, majordomo is a minor anachronism
nowadays. Just as majordomo largely replaced the ealier listserv
software, more recently mailman has mostly eclipsed majordomo.
However, to match current LPI topic areas, this tutorial will still
discuss majordomo.
As with most Linux tools, it is always useful to examine the manpages for any utilities discussed. Versions and switches might change between utility or kernel version, or with different Linux distributions. For more in depth information, the Linux Documentation Project has a variety of useful documents, especially its HOWTOs. See http://www.tldp.org/. A variety of books on Linux networking have been published; I have found O'Reilly's TCP/IP Network Administration, by Craig Hunt to be quite helpful (find whatever edition is most current when you read this).
A mailing list manager program is basically a local extension for a
mail transport program (MTA) such as sendmail. Basically, the MTA
running on a system passes off a set of address to the control of the
mailing list manager, and the mailing list manager modifies, processes
and perhaps remails the messages it receives. Some messages received
by a mailing list manager are messages meant for distribution to the
mailing list itself (perhaps needing to be verified for permission to
distribute to the list(s); other messages are control messages that
change the status of the mailing list, such as the subscription
options of a particular subscriber. A mailing list manager does not
perform mail deliver itself, but passes that function to its
supporting MTA.
As the introduction to this tutorial stated, Majordomo is not
currently the state-of-the-art choice for mailing lists. Rather, the
best choice for a new installation of a mailing list is probably
Mailman (http://www.list.org/). Majordomo, however, is still
perfectly functional, and is installed on many older systems that
continue to operate without problem (sometimes supporting lists that
have been operational for many years).
There is a wrinkle with Majordomo versions, however. Some years ago, a rewrite of the Majordomo 1.x series was started, called Majordomo2. Unfortunately, that rewrite fizzled out without ever reaching release status. While Majordomo2 (in a beta version) may be used in a very small number of systems, Majordomo 1.9.5 is the most recent stable version, and the version that will be discussed in this tutorial.
You can obtain an archive of the Majordomo software at
<http://www.greatcircle.com/majordomo/>. The most recent, and probably
final, stable version is 1.9.5. After unpacking a file named like
majordomo-1.94.5.tgz, be sure to read the file INSTALL carefully.
You will need to follow all the steps it describes to getting a
working Majordomo system. Building the system uses the usual make;
make install steps of most source installs, but also needs make
install-wrapper. The install can and should verify itself with a
command like cd /usr/local/majordomo-1.94.5; ./wrapper config-test
(the make install will provide you with details as a message).
Before building, you will need to modify the Makefile, and create
and/or modify majordomo.cf. The latter file can be copied from
sample.cf in the source distribution as a starting point. In the
Makefile, a number of environment variables are set, but the most
critical and subtle of these is probably W_GROUP. This is the
numeric gid of the group Majordomo will run under, almost always the
group daemon. The gid for daemon is 1 on most systems, but be
sure to check using:
$ id daemon uid=1(daemon) gid=1(daemon) groups=1(daemon)
Other variables in Makefile include PERL for the path to the
interpreter, and W_HOME for the location where Majordomo will be
installed.
Your new majordomo.cf file also needs to be edited before the make
isntall. Mostly the Perl variables that need to be modified appear
near the top of the file. Definitely adjust $whereami and
$homedir, and examine the others to make sure they are sensible.
The final step in installation is convincing Sendmail to talk with
Majordomo. Within the /etc/sendmail.cf file, this will involve a
line like:
OA/path/to/majordomo/majordomo.aliases
If you use the M4 processor to generate Sendmail configuration files, you can use a line like:
define(`ALIAS_FILE',`/etc/aliases,/path/to/majordomo/majordomo.aliases')
The sample majordomo.aliases contains some sample values:
majordomo: "|/usr/test/majordomo-1.94.5/wrapper majordomo" majordomo-owner: you owner-majordomo: you test: "|/usr/test/majordomo-1.94.5/wrapper resend -l test test-list" test-list: :include:/usr/test/majordomo-1.94.5/lists/test owner-test: you test-owner: you test-request: you
These, of course, need to be customized for your particular setup. In particular "you" means the name of the list administrator (who is not necessarily the overall system administrator).
The sample setup given above created a list called test, with
addresses for test-owner, test-request, etc. for administering the
list. Presumably, in real use you will want lists with other names.
To do that, do the following:
* Switch to the directory $listdir, as defined in majordomo.cf.
* Create a files called my-list-name and my-list-name.info (adjust
appropriately); chmod them to 664. The latter file contains an
introduction to the list.
* Create several aliases in your majordomo.aliases file, following
the pattern of the "test" examples. E.g.foo-owner,foo,foo-requestand so on.
* Send requests to "subscribe", "unsubscribe", "signoff", and so on
member of the list.
* Create an archive directory in the location specified by the
$filedirand$filedir_suffixvariables.
* Create a digest subdirectory under $digest_work_dir. Use the same
name as the digest list (example: test-digest).
* Make sure everything is owned by user majordomo, group
majordomo, and writable by both owner and group (i.e., mode 664
for files and mode 775 for directories).
* Issue a config <listname> <listname>.admin command to
Majordomo. This will cause it to create a default configuration file for the list, and send it back to you.
The short answer is that Sendmail is a Mail Transport Agent (MTA); it routes, modifies, and delivers mail message across heterogeneous mail systems. In a history somewhat parallel to that with mailing list software, Sendmail has a "permanent beta" version called Sendmail X that is intended as an upgrade/replacement for the stable Sendmail 8.x series; however, much as Mailman has largely supplanted Majordomo, several MTA have partially eclipsed Sendmail. The chief such new MTA is Postfix (http://www.postfix.org/), but Qmail (http://cr.yp.to/qmail.html) and Exim (http://www.exim.org/) are also relatively widely used. Nonetheless, Sendmail probably still remains, by a narrow margin, the most widely used MTA on Linux systems. As of September 16, 2005, the latest stable release of Sendmail was 8.13.5.
The long answer to what Sendmail is a very long answer indeed. Not just one, but many, books have been written on Sendmail. See http://www.sendmail.org/books.html for a list of available books. The most comprehensive of these is Sendmail, Third Edition by Bryan Costales with Eric Allman, Third Edition December 2002, ISBN: 1-56592-839-3. At 1232 pages, this book covers quite a lot more than this short tutorial can touch on.
While Sendmail in principle supports a number of mail transport
protocols such as UUCP, by far the most widely used one is Simple Mail
Transport Protocol (SMTP), which here includes Extended SMTP (ESMTP)
for enhanced MIME encoded message bodies. At heart, mail which is not
forward to other SMTP hosts is delivered to the local system, by
putting messages in local files. Local Mail User Agents (MUAs) read
messages that Sendmail (or another MTA) puts in local files (and often
also fetch mail using POP3 or IMAP), but generally call on Sendmail to
deliver outgoing messages. Some MUAs, however, themselves directly
communicate with SMTP servers (such as Sendmail instances, local or
remote) rather than placing messages in in the Sendmail queue for
later processing. Usually the Sendmail queue is in
/var/spool/mqueue/
First thing, obtain a copy of the current Sendmail software from
<http://www.sendmail.org/>, e.g. sendmail.8.13.5.tar.gz. Unpack it
as usual. Unlike many applications that use the make;make install
pattern, building Sendmail is peformed with sh Build. After the
initial build, cd to the cf/cf/ subdirectory; copy a suitable
*.mc file sendmail.mc; customize sendmail.mc; and run the
following to generate a sendmail.cf file:
$ m4 ../m4/cf.m4 sendmail.mc > sendmail.cf
You may also use the shortcut sh Build sendmail.cf. This may seem
mysterious, but what either of these commands do is generate an actual
Sendmail configuration from a more readable format using the M4 macro
processor. Actual sendmail.cf files, though editable ASCII, are
quite cryptic, and should only be modified by hand minimally.
Finally, copy the sendmail binary from a location like
obj.Linux.2.6.10-5-386.i686/sendmail/sendmail to its final location
(backup an old one if it exists), typically /usr/sbin/; and copy
your sendmail.cf file to /etc/mail/sendmail.cf. The latter can
also be peformed in the cf/cf/ subdirectory with sh Build
install-cf. You will probably need to su or sudo to obtain file
permissions for the relevant directories.
A number of utilities come with Sendmail: makemap, mailstats, etc.
Each corresponding directory has a README, and can be installed with
sh Build install run from the subdirectory.
sendmail.cf file
The main complexity, and the main function, of Sendmail is in its
sendmail.cf file. This configuration file contains some settings
for the Sendmail environment, but principally it contains patterns for
addresses to rewrite and/or deliver by certain mechanisms.
Two rewrite mechanism that may be configured are the genericstable
and virtusertable which let you map local users to and from external
addresses. For either mapping, you first create an aliases file as
plain text, e.g.:
david [email protected] root [email protected] [email protected] [email protected]
Or for incoming mail mapped to local accounts, e.g.:
[email protected] david [email protected] david [email protected] david @mail.gnosis.cx %[email protected] [email protected] owner%3 [email protected] error:5.7.0:550 Address invalid
To compile these aliases, use the makemap utility:
$ makemap dbm /etc/mail/virtusertable < inbound $ makemap hash /etc/mail/genericstable < outbound
Enabling use of these maps can be configured using M4 macros in
sendmail.cf (of in whatever configuration file you use).
DOMAIN(gnosis.cx)dnl FEATURE(`virtusertable', `dbm /etc/mail/virtusertable')dnl FEATURE(`genericstable', `hash /etc/mail/genericstable')dnl GENERICS_DOMAIN_FILE(`/etc/mail/generics-domains')dnl
A number of things are going on here. The DOMAIN macro indicates
that a file like cf/domain/gnosis.cx.m4 is used for additional
macros. The FEATURE macros enable use of the virtusertable and
genericstable. The GENERICS_DOMAIN_FILE macro defines the
domains that qualify for remapping for names in genericstable.
Rewriting will follow all the rules indicated. In test mode
(sendmail -bt) you can examine the rewriting that is performed for
specific addresses. For example, using genericstable, mail to the
local user david will be delivered to [email protected]
externally. Assuming localhost is defined in
/etc/mail/generics-domains, mail to david@localhost will go to the
same place.
In the other direction, mail coming in for [email protected] will
be rewritten and delivered to local user david. Multiple domains
can be manipulated by Sendmail at the same time, so
[email protected] will also be delivered locally. The full
power comes in some of the wildcard symbols. Any mail send to
mail.gnosis.cx that is not specifically directed to a local user
will be forwarded to the same username at external-host.com. But
that's a simple pattern. More interestingly, the %3 can be used to
expand multiple extra name information, so
[email protected] and [email protected] will
be delivered to local users owner-foo and owner-bar, respectively
(unless they undergo further rewriting). Presumably, these local
users might be mailing list processing systems or other automated
message handlers. As a special case, you can raise an error for a
given address rather than rewrite it further.
Really what we have looked at just scratches the surface of the rewriting rules you can add to Sendmail, but they give you an initial feel. Buy one of the large books on the topic to learn more details.
Sendmail can run in a number of modes. The most common mode is as a daemon that stays in the background and periodically. For example, running:
$ /usr/sbin/sendmail -bd -q10m
Tells Sendmail to run as a daemon, and check its queue every ten minutes. You can also run Sendmail a single time to process the queue at once, but not daemonize:
$ /usr/sbin/sendmail -q
As we mentioned above, Sendmail has a test mode to examine address rewriting rules, e.g. (from the Linux Network Administrators Guide, http://www.faqs.org/docs/linux_network/x15583.html):
$ /usr/sbin/sendmail -bt ADDRESS TEST MODE (ruleset 3 NOT automatically invoked) Enter <ruleset> <address> > 3,0 [email protected] rewrite: ruleset 3 input: isaac @ vstout . vbrew . com rewrite: ruleset 96 input: isaac < @ vstout . vbrew . com > rewrite: ruleset 96 returns: isaac < @ vstout . vbrew . com . > rewrite: ruleset 3 returns: isaac < @ vstout . vbrew . com . > rewrite: ruleset 0 input: isaac < @ vstout . vbrew . com . > rewrite: ruleset 199 input: isaac < @ vstout . vbrew . com . > rewrite: ruleset 199 returns: isaac < @ vstout . vbrew . com . > rewrite: ruleset 98 input: isaac < @ vstout . vbrew . com . > rewrite: ruleset 98 returns: isaac < @ vstout . vbrew . com . > rewrite: ruleset 198 input: isaac < @ vstout . vbrew . com . > rewrite: ruleset 198 returns: $# local $: isaac rewrite: ruleset 0 returns: $# local $: isaac
Procmail is a mail processor. That term probably means nothing at all to the unitiated, so we need some explanation. Basically, once Sendmail or another MTA has delivered mail to a local mailbox, you might use a MUA to process the mail in your inbox. You save some messages to various folders; you delete others; you forward other messages to various interested parties; you repl to others; and so on. Doing these tasks in an MUA is a manual and interactive process, and is potentially time consuming.
Procmail is a program that can automatically do these tasks for you whenever the required processing can be stated in a rule-driven way. Obviously, when you write back to your mother about her personal email, some personal attention is required; but for a large class of other messages, it is useful to specify in advance exactly what you would like to happen when a given message is received. The rules that can drive automated message handling might involve specific pattern-based header field, certain contents in a message body, or even calls out to more specific and specialized external programs such as statistical spam filters.
Procmail probably came installed with your Linux distribution. If not
you can obtain the source archive at <http://www.procmail.org/>. As
of this writing, the latest version is 3.22. You can also probably
install Procmail as a binary using the install system of your Linux
distribution (e.g. on Debian: apt-get install procmail). Building
from source is a straightforward make install. All Procmail needs
to operate is the procmail binary and a ~/.procmailrc
configuration file (or possibly a global /etc/procmailrc).
Beyond installing Procmail in the first place, you need to get your
local mail system to utilize Procmail. An older mechanism to process
mail through Procmail is to use a .forward file; this will still
often work on a per-user basis. Usually a user will create a file
~/.forward that contains something like:
|/usr/local/bin/procmail
This will pipe each incoming message to Procmail. However, a better
and more common way to utilize Procmail is to tell your MTA to talk
directly to Procmail in the first place. In Sendmail, this involved
enabling the local_procmail feature, e.g. put the following in your
sendmail.mc file:
FEATURE(`local_procmail', `/usr/bin/procmail', `procmail -Y -a $h -d $u')
Once Procmail is enabled at all, it needs a file ~/.procmailrc that
contains the set of rules it processes in handling a given message.
Procmail is not a daemon, but rather a text processing tool that
accepts exactly one email message at a time via STDIN.
At heart, Procmail is just a set of regular expression recipes. You
may also define environment variables in the same fashion as in a
shell script. Recipes are executed in order, but flags may be used to
execute a particular condition only if the prior condition is
satisfied (A), or is not satisfied (E). Some Procmail recipes are
delivery recipes, and others are non-delivery recipes; the former
terminates processing of a given message, unless the c flag is given
to explicitly continue processing. The most common action of a recipe
is probably to store a message in a named mailbox, but you may also
pipe a message to another program or forward the message to a list of
addresses.
A recipe usually starts with a lock (optionally with a specific lock file, otherwise it is done automatically) and some flags, followed by some rules, and then by exactly one action. I.e.:
:0 [flags] [ : [locallockfile] ] <zero or more conditions (one per line)> <exactly one action line>
Of particular note are the implied flag H to match the header, and
B to match the body. Patterns normally are case-insensitive, but the
D flag can force case-sensitive matching.
If a condition begins with *, everything after that character is an
egrep expression. Otherwise, if a line starts with < or > it
checks the size of a message as smaller or larger than a given number
of bytes. The $ prefix allows shell substitutions in
An action that is simply a file name saves a message to that mailbox.
Use the special pseudo-file /dev/null to delete a message. A pipe
character (|) passes the message to another program, such as the
digest-splitting utility formail that is distributed with Procmail.
The exclamation prefix (!) forwards a message as an action (but
negates a condition in a rule). Some examples:
:0: * ^Subject:.*Digest # split digests and save parts * ^From:.*foo-digest |formail +1 -ds cat >>mailing_lists_mailbox :0: * !(To|Cc).*[email protected] # my main account here * !(To|Cc).*[email protected] # I get mail from here * !From.*gnosis\.cx # I trust gnosis not to spam * !From.*list.*@ # don't trash mailing lists * !From.*good-buddy # sometimes Bcc's me mail spam :0: * ^Subject.*[MY-LIST] # redistribute MY-LIST messages ! [email protected], [email protected], [email protected] :0: * ^Cc.*[email protected] # save to both inbox and JOE mbox { :0 c $DEFAULT :0 JOE }
NNTP is a nice protocol for "pull" distribution of messages to any users who are interested in a given topic. The Usenet is a large collection of "newsgroups", on thousands of different topics, which distribute messages via NNTP. Being a pull protocol, and NNTP server gathers the current message available from a decentralized network of servers, selecting only those newsgroups that the site administrator choooses to include. When a new message is posted to a given newsgroup, it propogates non-hierarchically from the server the user directly connects to all the other servers on the internet interested in subscribing to that particular newsgroup.
From an end user perspective, a mailing list can appear very similar to a newsgroup. In either case, the user composes and posts messages, and reads messages written by other people. In the ancient days of the Usenet and the Internet, mailing lists were not as capable of presenting discussion topics in a "threaded" fashion as newsgroups do automatically. But for a number of years, mail clients have done a good job of inferring the discussion threads within mailing lists. The main difference between newsgroups and mailing lists is in their underlying network protocol. A mailing list still relies on one centralized mail server that accepts all the messages destined for a particular list, and distributes that message via email to all users who have indicated and interest (and have been approved, either by automatic or human moderated subscription mechanisms). In contrast, NNTP connects every node to every other one without relying on a central server; each NNTP server simply talks to the other servers "nearby", and rather rapidly, this reaches the whole world.
InterNetNews (INN) is an NNTP server that was first written in 1992, and has been actively maintained since then. As of this writing, INN is at version 2.4.1. The home page for INN includes releases and documentation and is at <http://www.isc.org/index.pl?/sw/inn/>.
After obtaining and unpacking the current source release, builing INN
is a straightforward ./configure; make; make install sequence. To
build INN, you will need to have Perl and yacc (or bison) installed.
This will create a number of files, mostly in the /usr/local/news/
directory (which you probably do not have if INN has not been
installed previously).
Before running the innd daemon (as user news), you will want to
modify a number of configuration files. The full details will not fit
in this tutorial section, but a longer tutorial on the full set of
files needing attention (albeit being older, it is for SCO Unix, not
for Linux; but most details remain the same) can be found at
<http://www.kozubik.com/published/inn_tutorial.txt>. Many of the
permissions and quota issues will be handled by the make system, but
you might want to double check these configurations.
A file to pay particular attention to is the quota setup in
/usr/local/news/etc/storage.conf. This controls which newsgroups
are subscribed to, and how much history from each newsgroup to
maintain. Once the quota is reached, older messages are purged from a
given newsgroup (on the local server, not from Usenet as a whole).
For example, it might contain:
method cnfs {
newsgroups: alt.binaries.*
class: 1
size: 0,1000000
options: BINARIES
}
method cnfs {
newsgroups: *
class: 2
size: 0,100000
options: NOTBINRY
}
The "class" value specifies the order in which different rules are evaluated.
Once all the various configuration files are tweaked, just running
innd as a daemon (probably launched for an initialization script)
monitors the upstream servers configured by /usr/local/news/etc/innfeed.conf,
/usr/local/news/etc/incoming.conf and /usr/local/news/etc/newsfeeds.