Spam Vigilante - Mail Filter Virtual Appliance

Table of Contents

Sections

1. Introduction
2. Installation
3. Spam Viewer
4. Updates

Appendices

A. Integration Notes
B. Important Files
C. Software Used
D. References
E. Disclaimer

Section 1 - Introduction

This virtual machine mail filter appliance is designed to give small businesses and organizations the ability to deploy spam and virus checking for their e-mail communications in only minutes.  It does this work by acting as a preprocessor, retrieving and scanning all incoming mail and then forwarding it to its final destination.  It offers the unique benefits of allowing users to view and delete quarantined messages through a custom, easy-to-use web interface and a level of integration with Microsoft Exchange via LDAP, if that server is also being used.

Open source and free software is used throughout, so no licensing costs are incurred.

The mail filter is not the mail server that users will access directly. Because there is no user interaction, its deployment can be completely transparent, and no user account details need to reside on the appliance, allowing it to be integrated equally well into both new and existing mail infrastructures and to be compatible with any mail system which can accept SMTP connections (virtually any Internet-capable mail system).

Section 2 - Installation

Choosing a Layout:

The mail filter appliance is not a mail server in the traditional sense of the term.  It is intended as a complement to an existing service, supplementing it by pre-filtering virus-infected or spam messages, and processing them as directed.  By default this means marking spam messages, and quarantining viruses.  Though no traditional mail server is included in this appliance, if it is also required, one may wish to look into hMailServer, an excellent open source mail server for Windows operating systems, Postfix, the SMTP server used by this appliance, and Dovecot, an open source IMAP/POP3 mail server for UNIX-like operating systems, including Linux.  Links to all three are provided in Appendix D.

The mail filter is designed to operate in two primary modes, either as a standard SMTP server, shown in Figure 1.1, or as a "fetching" mail server, shown in Figure 1.2.

SMTP is the traditional form of mail server operation.  When a message is sent by another party, their SMTP server consults DNS to find the appropriate mail exchange (MX) records for the recipient -- in this case, the mail filter appliance.  The sending server connects to the recipient server and submits the message, which is delivered to a user's mailbox, completing the process.  The mail filter appliance works in this chain as a proxy, receiving the incoming message from SMTP, processing it, and then forwarding it again to the "real" SMTP server which contains the user's mailbox.  This approach requires registering the IP address and hostname of your appliance in DNS, so that other parties on the Internet can locate it, and a high-level of availability from your equipment, so messages cannot be missed or bounced.  It is best suited for organizations that need absolute control over all parts of their mail system and that can provide the level of maintenance and expertise needed to administer it.

By contrast, the fetching layout for the mail appliance does not require registering DNS or allowing inbound connections from the Internet.  Instead, a hosting provider is used to host the public SMTP server and an e-mail tool is used to retrieve the mail from the hosting company's server using POP3 or IMAP4, similar to a user checking mail with a mail client.  The messages are downloaded and injected into the appliance's SMTP server for processing.  After that, the chain is identical to the SMTP layout; the mail is forwarded to the "real" mail server on the network and is delivered into the receiving user's mailbox.  This layout is ideal for many small businesses and organizations, as it allows a high level of control over e-mail services, while providing protection against incoming mail being bounced because of  a local hardware or connection failure.  That is important because many organizations do not have the fully redundant hardware, power, and Internet connectivity that a quality hosting provider can supply.


Figure 1.1 - SMTP Layout

Figure 1.2 - Fetchmail Layout

Starting:

Once you've chosen a layout, to begin installation simply boot the virtual machine.  When presented with the login prompt, login using the username 'root' and the password 'password', and a setup wizard will launch.  Below is an example setup session using the fetchmail layout.  In the example, the internal network has a domain name of 'company.local' and the domain used for mail (hosted by a hosting provider) is 'company.com'.  The internal network in 192.168.100.0.  The final destination server for the processed messages is named mail.company.local and is running Microsoft Exchange.  Default values are shown by the wizard within brackets ("[]") and examples within parentheses ("()").  Additional comments are added where needed.

Network configuration:

  Enter the fully qualified hostname
  (eg, mailfilter.company.com) []: fetcher.company.local

If using SMTP-mode, it is best to enter a static IP address.
Fetchmail mode can use either.

Use DHCP to configure network settings? (Yes/No) [no]:

  Enter the IP address to use
  (eg, 192.168.100.10) []: 192.168.100.2

  Enter the subnet mask to use
  (eg, 255.255.255.0) []: 255.255.255.0

  Enter the address of the default gateway
  (eg, 192.168.100.1) []: 192.168.100.1

  Enter the address of the primary DNS server
  (eg, 192.168.100.1) []: 192.168.100.1

  Enter the address of the secondary DNS server
  (eg, 192.168.100.2) (blank for none) []:
Fetchmail configuration:

  Use fetchmail or use as SMTP server (fetchmail/smtp) [fetchmail] ?
In a deployment with many users it is quickest to enter the information
into fetchmail.cf directly. However, it can still be useful to enter an
account or two as is done here for use as examples, or as test accounts
before full deployment.

Would you like to enter account information to be used by fetchmail?
  If there are more than a few accounts it is recommended to edit the
  fetchmail configuration file (/usr/local/etc/fetchmail.cf)
  directly (Yes/No) [no]: yes
Using this option means no mail will ever by deleted from the
hosting mail server. It corresponds to the fetchmail option "keep."
After the server has been tested and shown to be operating correctly,
you should edit fetchmail.cf to remove "keep" from the accounts,
else the mailboxes on the remote server will eventually fill.
Would you like to keep messages on the servers after fetching?
  Strongly recommended when first configuring a system.  (Yes/No) [yes]:

  Enter a server to poll (blank to quit): mail.company.com

  Enter a username/address for this server (blank to quit): user_one@company.com

  Enter a password for this account: password_one

  Enter the local address to take delivery: user_one@company.local

  Enter a username/address for this server (blank to quit): user_two@company.com

  Enter a password for this account: password_two

  Enter the local address to take delivery: user_two@company.local

  Enter a username/address for this server (blank to quit):

  Enter a server to poll (blank to quit):

E-mail setup:
The answers here reflect a common layout when using Microsoft Small Business
Server, where the internal domain is distinguished from the public domain by
the use of the .local suffix. This eases DNS maintenance.  This arrangement may not
be the case in your environment, however.  There is no requirement that the external
mail domain and the internal one be different.  Even in this instance, delivery would be
 unaffected if company.com were entered in both fields, but using .local is convenient
to show the separation and to make items more easily distinguishable in logs.
External mail domain (eg, redbudcomputer.com) []: company.com

  Internal mail domain (eg, redbudcomputer.local).
  Very often this is the same as the external []: company.local
This setting is the hostname of the "real" mail server on your
network that should take delivery of all messages. Make sure
it is entered accurately and that it is reachable by the appliance.
If it is not, mail could be bounced.
Enter the full name or address of the mail server to take
  final delivery after filtering is complete []: mail.company.local

  Enter a comma separated list of domains this server
  should service / relay ['company.com', 'company.local']:
The default value of this field will be 'host' if using
fetchmail-mode and 'subnet' if using SMTP-mode. 
If this appliance is also the SMTP server which will send
outgoing messages for your clients, at the least it must be
set to 'subnet'.  If your network consists of more than one
subnet, enter a comma separated list of all the
networks from which it will accept connections.
Enter 'host,' subnet,' or a comma separated list of
  networks (eg, 192.168.100.0/24) from which this server
  should allow clients.  The default option is usually best unless
  one is sure they have other requirements ['host']:

Spam configuration:
For all the destiny options below: 'D_PASS' means the message
is delivered as usual. In the case of spam, new headers may be added
and the subject of the message changed, allowing mail clients to filter
on the values. 'D_BOUNCE' means a bounce notification is returned
to the sender and the message is quarantined. 'D_DISCARD' means
the message is quarantined and no notification is sent to the sender.
Spam message destiny (D_PASS, D_BOUNCE, or D_DISCARD).
  It is safest to use D_PASS, and allow the clients to filter
  on the X-Spam-Status or X-Spam-Level headers. [D_PASS]:

  Virus message destiny (D_PASS, D_BOUNCE, or D_DISCARD).
  It is strongly recommended to use D_BOUNCE or D_DISCARD so
  that infected messages cannot be delivered to clients. [D_BOUNCE]:

  Banned attachement message destiny (D_PASS, D_BOUNCE, or D_DISCARD).
  D_BOUNCE or D_DISCARD should be used so disallowed attachments (eg, .exe)
  are not delivered to clients. [D_BOUNCE]:

  Bad header message destiny (D_PASS, D_BOUNCE, or D_DISCARD).
  Leave this at the default if delivering a message with a malformed
  header is OK. [D_PASS]:

  Spam level - the score required to mark a message as spam
  (X-Spam-Status: Yes) and trigger corresponding action. [6.31]:
Many organizations find it valuable to ban certain types of attachments
regardless of whether they contain a known virus.  Common examples
are .exe and .vbs, both of which are blocked by default.  If encapsulating
a banned file type within an archive, such as a .zip or .gz, should allow it
to pass, set this to "yes." If banned attachment types should never be
allowed, leave this at the default "no."
Allow banned attachments if they are inside a .zip file?
  Eg, if .exe attachments are banned, they can still be received
  if the are packaged inside a .zip attachment.  The full list of affected
  archive types is: zip, bz2, gz, tgz, tbz. (Yes/No) [no]: yes

  SA maximum mail body size (in bytes). Messages larger than this
  size will be scanned for viruses, but not for spam. [524288]:

  Maximum size of the quarantine mail folder (in MB).
  When the quarantine folder reaches this size, the oldest
  messages are deleted [500]:
This enables additional spam checking rules beyond the base set
used by SpamAssassin. The rules are updated automatically and additional
rules can be added by editing /etc/rulesdujour/config.  In general, the more
rules are available, the better the spam scoring functions. The additional
rules used in the appliance are TRIPWIRE, SARE_EVILNUMBERS0,
SARE_RANDOM, and SARE_ADULT.
Use extra spam checking rules from RulesDuJour?  Recommended
  unless the server is under a high load. (Yes/No) [yes]:

Integration configuration:
This chooses whether to enable the web interface for viewing quarantined
messages.  The viewer allows a user to see and delete quarantined messages,
and an administrator to see messages for any user and to clear the
quarantine folder. See Section 3 for more information about the web viewer.
Enable web interface for viewing quarantined
  messages?  The URL is https://<server>/viewer/ (Yes/No) [yes]:
Exchange integration takes two forms. The first is Postfix integration:
the Exchange server is periodically queried to retrieve a list of mail accounts,
and an allowed recipients list is constructed from the results. The second is
viewer integration: users can login to the web viewer (if enabled)
using their Exchange e-mail address and password.
Use Microsoft Exchange recipient integration?
  This provides automated construction of an allowed recipients
  map by querying the Exchange server.  If you selected to enable
  the web viewer, it allows users to login to it using their Exchange
  address and password (Yes/No) [yes]:

  By default the integration tools will attempt to establish
  a secure connection to the Exchange server, and if that fails
  will attempt an insecure connection.  If it is never allowable
  to try an insecure connection you should change the value of
  LDAP_ALLOW_INSECURE in /usr/home/spamviewer/config.py to 'False'.

  Enter the hostname of the Exchange-enabled
  Active Directory server to connect to []: mail.company.local
See Appendix B for tips on creating a suitable account.
Enter an Active Directory username to use to connect.
  The username can be in the form of an LDAP DN or @.
  The user only needs read access to the part of directory to be scanned
  for accounts.  Do not use a privileged account! [] sync@company.local

  Enter the password for the account []: password
In a standard Active Directory installation, with a domain
name such as 'company.local,' the root of the Active Directory tree will be
'DC=company,DC=local'. If users are throughout the tree one should use
the root, tough if the directory is very large, queries may be slow or cause
a high load on the directory server. If all users are centralized, a more specific
location can be used. Common user locations are
'CN=Users,DC=company,DC=local' and, in the case of Small Business Server,
'OU=MyBusiness,DC=company,DC=local'. Refer to
http://support.microsoft.com/kb/q224543/ for information about finding
the DN of objects within the directory.
Enter the base DN of the topmost container to scan.
  This can be the top of the domain such as "DC=company,DC=local"
  or a sub-container like "CN=Users,DC=company,DC=local" []: OU=MyBusiness,DC=company,DC=local

Review:

Network configuration:
  Hostname: fetcher.company.local
  Use DHCP: no
  IP Address: 192.168.100.2
  Subnet Mask: 255.255.255.0
  Default Gateway: 192.168.100.1
  Primary DNS: 192.168.100.1
  Secondary DNS:

Mail configuration:
  Destination mail server: mail.company.local
  Fetchmail or SMTP: fetchmail
    mail.company.com:
      user_one@company.com   password_one   user_one@company.local
      user_two@company.com   password_two   user_two@company.local
  Ext. mail domain: company.com
  Int. mail domain: company.local

  Spam destiny: D_PASS
  Banned destiny: D_BOUNCE
  Bad header destiny: D_PASS
  Virus destiny: D_BOUNCE
  Spam level: 6.31
  Allow banned inside zip?: yes
  Max message size to scan for spam: 524288
  Use extra spam rules?: yes
  Allowed network: host
  Relay domain: company.com
  Relay domain: company.local

Integration:
  Use web quarantine viewer?: yes
  Use Exchange: yes
  Exchange server: mail.company.local
  Active Directory user: sync@company.local
  Active Directory pw: password
  Active Directory base DN: OU=MyBusiness,DC=company,DC=local

Configuration OK? (Yes/No) [Yes]:

Status messages will be displayed as the services are reconfigured.
If Exchange integration was chosen and the error message
'{'desc': "Can't contact LDAP server"},' is printed, it likely means
either the server name or Active Directory account details were
not correct. Edit them directly in /usr/home/spamviewer/config.py,
or re-run the setup script at /usr/local/etc/conftemplates/spam_configure.py.
All previous answers will be saved.
...

It is recommended that you restart your system soon with
'shutdown -r now' to engage all of the configuration changes. Before 
restarting, however, this is also an excellent time to change the root
password, using 'passwd,' and enter any additional fetchmail configuration 
information before rebooting.

To reconfigure your machine at any time simply execute: 'touch /reconfigure'
and reboot!

Good hunting!

That is it!  Configuration of an SMTP style server is very similar, except for entering fetchmail account details.  Likewise, entering information for a server without Exchange integration is the similar, except that the sections dealing with the Active Directory user name, base DN, and other details are skipped.
 

To activate all the settings, reboot the appliance by executing 'shutdown -r now'.  During boot, a message may be shown stating that the virus database is older than seven days.  This is not an error.  It will be automatically updated, and kept up to date, from now on since the network settings have been configured.  As soon as it has rebooted, the appliance is ready to handle mail.  Try sending and receiving test messages to confirm, and in particular examine the headers of incoming messages for an entry such as "X-Virus-Scanned: amavisd-new at company.com."  If any problems are encountered, try adjusting the configuration through the setup wizard by running "python /usr/local/etc/conftemplates/spam_configure.py" or by executing "touch /reconfigure" and rebooting.  In the latter case, the setup wizard will present itself again at login.

Note: Though the appliance has SSH access, configuring over SSH can be troublesome as the network interfaces are restarted during the final stage.

Section 3 - The Spam Viewer

Introduction:

The Spam View web application was specifically made for this appliance.  If enabled, the Spam Viewer can be used to view any quarantined messages through a web interface.  The administrator account ('vadmin') can view the messages for any user.  Users may also login to view their own messages.

Login operates in two modes, LDAP login and local login.  The default login method is dependent on whether Exchange integration was chosen.  If it was selected, LDAP login is used, allowing users to login to the viewer using their Exchange e-mail account and password.  The standard login method requires that user accounts be created by 'vadmin' before those may login. 

Note: The login method may be selected manually by setting the value of the LOGIN_METHOD variable in config.py.

Administration

To begin administering the viewer, login with the username 'vadmin' and the password 'password'.  A screen similar to the one in Figure 2.1 will open.

Note: If the LDAP login method is being used, the "Add User" link will not be present.


Figure 2.1 - Administration Console

From this console you can change the password for the 'vadmin' user, clear the quarantine/spam folder, and manage user accounts for the viewer.

User Accounts

Before users can login to view their messages, accounts must exist for them in the viewer's database.  User accounts in the system consist of two primary parts: the username and a list of associated e-mail addresses.  Each e-mail address may only be associated with one user, but each user can have multiple e-mail addresses. This is particularly useful with a fetchmail-style layout where a single user can be receiving mail from multiple external addresses (e.g. where mail for user@domain_one.com and for user@domain_two.com is being delivered to user@company.com).  Each user can only view messages addressed to an account contained within their list of allowed addresses.

Adding Users

If user accounts are being maintained locally rather than using the LDAP login method, user accounts will need to be added manually.  Selecting 'Add User' from the Administration Console will show a page similar to the one if Figure 2.2.  Enter the username and one e-mail address per line in the 'Address' box.  Press 'Save' to store the user.
 

Figure 2.2- New User

If the LDAP login method is being used instead, users accounts are drawn from the remote server.  They are updated daily by the script '/usr/local/sbin/maintain-spamviewer.sh' so that new user accounts will be found automatically and e-mail addresses found in the directory are assigned to the users.  You may also execute this script manually to perform an update immediately.  By default, accounts are not altered once they are created.  To perform a full update that clears existing user information entirely and re-draws it from the directory server execute '/usr/local/sbin/maintain-spamviewer.sh' with the '-c' option.

Editing a User

The Users page, accessible from the Administration Console, will present a list of all the users in the system similar to Figure 2.3.


Figure 2.3 - Users

Clicking on "Delete" will remove the user account and clicking "Messages" will allow you to view that user's message list.  Clicking on the username in the Users page will allow editing of the user's details.  From here, you can change the e-mail addresses associated with an account and, if the account information is local, the user's password.  Figure 2.4 shows the Edit User page.

Note: If you are using LDAP and the list of users is empty, you may wish to execute the '/usr/local/sbin/maintain-spamviewer.sh' script.  This script is executed periodically, but if you have recently installed the system it may not have been executed yet.


Figure 2.4 - Edit User

Viewing Messages

The message view shows the basic message details, such as the subject and the sender.  The messages are sorted by date.  It also displays the reason for quarantine -- VIRUS, SPAM, BADH[eader], or BANNED.  Messages may be filtered using the form in the upper corner.  The search phrase is used to search the Subject, From, and To fields, but does not search the message body. 

Clicking upon the subject will display the contents of the message in a safe, plain-text view.  The Download link allows the original message to be retrieved in the message/rfc822 MIME-type.  By default, the Download link is not available for messages infected with a virus but is for all other types, including those with banned attachments.  This can be changed by altering the value of the ALLOWED_DOWNLOAD_TYPES variable in config.py.

The Delete All link will remove all quarantined messages for the user.

Added in an update to the original package, it is also possible to release a message from the quarantine to resume normal delivery.  This option takes the form of a red right-arrow (not pictured) next to the download link.


Figure 2.5 - Messages

Section 4 - Updates

Updates and fixes:

As new features, updates, or bug fixes are released, downloads and information will be made available at http://www.redbudcomputer.com/downloads.htm.

Appendix A - Integration Notes

In order to integrate the Postfix recipients list and Spam Viewer user account database with information from Exchange, an account needs to be available on the Exchange/Active Directory server that can be used to retrieve the information.  This user should have no special privileges, as all that is required is read access on the items within the parts of the directory tree that are to be searched.

One option is to create a dedicated user account so that access can be audited and controlled.  A sample structure is illustrated in Figure B.1, where the user account has been added to a group with restricted login privileges.  User Rights Assignments within a Group Policy Object are used to prevent any member of the group from logging in locally to any machines in the Active Directory domain.  The key properties of the restriction are listed in Figure B.2.  The group should not be added to the the "Deny access to this computer from the network" property, or the user will not be able to perform the necessary queries.


Figure B.1 - GPO

Figure B.2 - GPO Style

Additional security can be added by controlling the allowed login hours for the user to those used by the automated task and by implementing auditing of user logon events. Both topics, and instructions on how to create the group policy object are beyond the scope of this documentation.  Please refer to http://www.microsoft.com/windowsserver2003/technologies/management/grouppolicy/default.mspx (group policy), http://support.microsoft.com/?kbid=814595 (auditing), and http://technet2.microsoft.com/WindowsServer/en/Library/f94917d2-660a-48a0-a1e6-9a67252c6dd71033.mspx (logon hours) for more information.

Appendix B - Important Files

The locations of important configuration files are listed below.

Amavis (Mail Scanner):
/usr/local/etc/amavisd.conf
/var/virusmails

Apache (Web Server):
/usr/local/etc/apache2/http.conf
/usr/local/etc/apache2/ssl.conf
/usr/local/etc/apache2/Includes/spamviewer.conf

ClamAV (Anti-virus):
/usr/local/etc/clamd.conf
/usr/local/etc/freshclam.conf

Exchange Integration:
/usr/home/spamviewer/config.py

Fetchmail (Mail Agent):
/usr/local/etc/fetchmail.cf

Postfix (SMTP):
/usr/local/etc/postfix/main.cf
/usr/local/etc/postfix/master.cf
/usr/local/etc/postfix/transport
/usr/local/etc/postfix/exchange_recipients (optional)

SpamAssassin (Spam Processing):
/usr/local/etc/mail/spamassassin/local.cf
/etc/rulesdujour/config

Spam Viewer:
/usr/home/spamviewer/config.py

Configuration Template Files:
/usr/local/etc/conftemplates/

Appendix C - Software Used

amavisd-new 2.4.1 GNU Public License
Apache 2.0.58 Apache License
clamav 0.88.2 GNU Public License
Fetchmail 6.3.2 GNU Public License
FreeBSD 6.1 BSD License
mod_python 3.2.8 Apache License v2.0
OpenSSH 4.2 BSD License
OpenSSL 0.9.7e OpenSSL (Apache-style) License
Perl 5.8.8 Artistic License
Postfix 2.2.9 IBM Public License v1.0
pysqlite 2.1.3 pysqlite Open Source License
Python 2.4.2 Python 2.4 License
python-ldap 2.2.0 python-ldap (Python-style) License v1.1
Quixote 2.4 CNRI Open Source License
Razor 2.81 Artistic License
session2 MIT License
SpamAssassin 3.1.1 Apache License
Spam Viewer Creative Commons Attribution License v2.5

Appendix D - References

Appendix E - Disclaimer

The information in this document is provided "as-is."  No warranty is expressed or implied and no guarantees are made to its accuracy.  Any loss of information, data, or revenue resulting from the use of the information contained in this document is not the responsibility of the author.  Unless specifically stated, the author of this document does not claim to be the original author of any works or products used in this appliance.  All copyrights are the property of their respective owners.

 

Copyright (C) Thomas E Lackey 2006
Licensed under the Creative Commons Attribution License v2.5