Setting up IMAP

Introduction/Theory

As far as I know, there are currently three open-source IMAP servers available: University of Washington (the one that typically ships with Linux distros, and written by the father of IMAP Mark Crispin), Cyrus IMAP, and Courier IMAP. Commercial servers include Lotus Notes, Novell GroupWise, Microsoft Exchange, Stalker Software's CommuniGate Pro, ISOCOR's N-Plex Global, Gordano's NTMail (formerly Internet Shopper), IPSwitch's IMail, QUALCOMM's Eudora WorldMail Server, Rockliffe's MailSite and Vintra Systems' Mail Server Professional.

In the absolute, IMAP is a great improvement over POP, but I have yet to find an IMAP client that supports disconnected mode easily for average users. Considering the number of users with laptops, it is strange that no company has yet solved this issue fully.

IMAP considers three possible ways to work with an IMAP server: online (no e-mail is downloaded into the client host's hard disk), off-line (all e-mail is downloaded in the client host's hard disk and deleted from the server, ie. POP3-style), and disconnected (e-mail is kept synchronized on server and client hosts).

There are currently two ways to manage e-mails on a server:

Each has its strength and weakness: Using a single ASCII file for a mailbox makes it easy to restore files from tapes and grep for a pattern; the drawback is that you should remember to archive e-mails often to keep those mailbox files from growing too big and force IMAP to parse too many e-mails.
On the other hand, keeping each e-mail into a separate file means more I/O and you must also pay attention to the number of i-nodes available for the file system where /var/spool/imap/user lives. Mark Crispin explains that the best way is actually to dump both methods, and use a DBMS, like Exchange does.

Several file formats exist to build e-mail mailboxes (as explained by Mark Crispin): Unix (traditional format; each e-mail is isolated by putting "From " on the first line of each e-mail; the file must be read each time the mailbox is opened, and line ends must be converted from newline to CRLF), mbx (current preferred format; messages are stored in CRLF; allows shared access), mx (experimental; each mailbox has a .mxindex file which holds message status and identifiers; stored in CRLF; not very efficient).

Note: When connecting to UoW IMAPd through Xinetd, Xinetd seems to try to connect to TCP port 113 (Identd) on the calling client. If the client is running a firewall (eg. to protect it because it is also connected to the Net through ADSL), there will be a delay until Xinet times out. If the firewall does not handle calls to TCP 113, one solution I found is to comment out the following lines in /etc/xinetd.d/imap:

#       log_on_success          += DURATION USERID
#       log_on_failure          += USERID

 

UoW IMAP

  1. Install the latest and greatest RPM (eg. imap-4.7c2-2.i386.rpm). You might want to install extra utilities, eg. imap-utils-4.7c2-2.i386.rpm
  2. Check that /etc/inetd.conf points to the right files, and run killall -HUP inetd after updating if necessary
    pop-3 stream tcp nowait root /usr/sbin/tcpd ipop3d
    imap stream tcp nowait root /usr/sbin/tcpd imapd
  3. Create a first user through eg. useradd jdoe ; passwd jdoe
  4. To keep all user e-mails inside their home directory, create an empty mbox file at the top level (touch /home/jdoe/mbox), and create a sub-directory mail in users' home directory (mkdir /home/jdoe/mail) to keep the IMAP clients from showing all files and directories that exist at the top level of their home directory, eg. .Xdefaults, .bash_profile, etc.
  5. (optional) I also like to create Outbox, Old | Inbox, Old | Outbox sub-directories in users' home directory so they can start organizing their e-mails without having to create directories themselves: mkdir -p /home/jdoe/mail/Outbox /home/mail/Old/ /home/mail/Old/

    Mailboxes must contain the following original system e-mail:

    		From MAILER-DAEMON Sun Nov 12 17:06:32 2000
		From: Mail System Internal Data 
		Date: 12 Nov 2000 17:06:32 +0100
		Subject: DON'T DELETE THIS MESSAGE -- FOLDER INTERNAL DATA
		Message-ID: <974045192@linux.acme.com>
		X-IMAP: 0974045178 0000000001
		Status: RO
		
		This text is part of the internal format of your mail folder, and is not
		a real message.  It is created automatically by the mail system software.
		If deleted, important folder data will be lost, and it will be re-created
		with the data reset to initial values.

    Put the above into a template, and cat init-imap.txt > /home/jdoe/mail/Outbox ; cat init-imap.txt > /home/jdoe/mail/Old/Inbox ; cat init-imap.txt > /home/jdoe/mail/Old/Outbox ; chown -R jdoe:jdoe /home/mail/ ; chmod -R 0600 /home/jdoe/mbox /home/jdoe/mail/

  6. Set up an IMAP client to point to your UoW server: Here, the mailbox home directory should be ~/mail

Cyrus IMAP

Cyrus IMAP originated from Carnegie Mellon University, and is also the base for commercial IMAP servers like Netscape, Mirapoint, or MessagingDirect.

Starting with release 1.6.x, Cyrus uses the SASL library to externalize authentication. It supports /etc/passwd, PAM, sasldb, Kerberos. The RPM that I found on RPMFind was built by RedHat and set up to use PAM -> /etc/shadow.

Conceptually, SASL appears to be similar to PAM, ie. it lets developers delegate authentication so as to allow administrators change authentication methods without recompiling applications. Considering that the RPM packages used here were compiled by RedHat to have PAM authenticate against /etc/shadow, we end up not only with a double indirection (SASL -> PAM), but we don't even make use of SASL's password DB (use saslpasswd to set a password for a Cyrus IMAP e-mail account, and have them saved in binary format in /etc/sasldb.)

Among things that I don't like about Cyrus, are its lack of support for Procmail (it uses Cyrus' own MDA Sieve), and it won't allow you to create other top-level mailboxes along Inbox, eg. Outbox. All user-created mailboxes must reside inside Inbox. It doesn't make sense to users to have Inbox | Outbox, Inbox | Old | Inbox, etc. User mail files are usually located in /var/spool/imap. The user's Inbox is user.jdoe, and the name of other folders is appended, ie. user.jdoe.outbox, user.jdoe.drafts, etc.

Installation and configuration

Unless really necessary, I much prefer to take advantage of RPM to install applications. Since Cyrus 2 is still under development, I chose to use the current stable release, 1.6.x:
  1. Install the SASL and IMAP packages. Those RPM packages will also create the account (cyrus) that Cyrus IMAP uses to run, and add it to the mail group:
    rpm -Uvh cyrus-sasl-1.5.11-2.i386.rpm
    rpm -Uvh cyrus-imapd-1.6.19-2.i386.rpm
  2. Use passwd cyrus to set a password for the cyrus:mail account under which Cyrus IMAP runs
  3. Launch the administration utility to create a first user account: cyradm -u cyrus localhost, followed by cm user.jdoe
  4. Exit cyradm, and create the ad hoc jdoe account through useradd jdoe ; passwd jdoe. If those users should only use IMAP but not be allowed to have a shell account to the IMAP server, you might want to run usermod -s /bin/false jdoe. If you want to allow them to change their password, usermod -s /usr/bin/passwd jdoe.
  5. The packages were compiled to have SASL use PAM to authenticate users against /etc/shadow. "Managing IMAP" talks about a utility called pwcheck which is necessary to have SASL authenticate users against /etc/shadow, but it wasn't included in the packages. I guess the use of PAM makes pwcheck unnecessary.
    Until I find a better solution (Kerberos? SASL and /etc/sasldb? LDAP?), users could not log on until IMAPd was allowed to read /etc/shadow: usermod -G root cyrus ; chmod 0440 /etc/shadow
  6. In /etc/inetd.conf, check that inetd actually launches Cyrus when someone connects to port 143. If not, update and run killall -HUP inetd:
    pop3 stream tcp nowait cyrus /usr/cyrus/bin/pop3d pop3d
    imap stream tcp nowait cyrus /usr/cyrus/bin/imapd imapd
  7. Check that IMAP works through
    imtest -u cyrus -m login imap.acme.com
  8. If using PostFix as your MTA, edit /etc/postfix/main.cf to add mailbox_transport = cyrus. /etc/postfix/master.cf should already one line dealing with Cyrus IMAP.

Resources

Courier IMAP

(Feb 15, 2002) I added this section while trying out the latest and greatest of PhpGroupware which only supports Courier IMAP or Qmail for POP3 access. Note that Courier also exists as a full MTA + IMAP/POP package, but here, we'll use the IMAP/POP package.

A major difference with the other two IMAP servers is that Courier only uses Maildirs (ie. supported by Qmail, and also Exim and Postfix if configured that way) instead of the traditional mailbox files (eg. /var/spool/mail). As Sendmail doesn't support Maildirs, you'll have to use one of the MTAs above to work with Courier.

Creating an RPM

  1. Log on as a non-root user, and cd to its home directory
  2. Create the following directories: mkdir -p $HOME/rpm/SOURCES $HOME/rpm/SPECS $HOME/rpm/BUILD $HOME/rpm/SRPMS $HOME/rpm/RPMS/i386
  3. echo "%_topdir    $HOME/rpm" >> $HOME/.rpmmacros
  4. Download the latest Courier IMAP tarball, copy it in eg. $HOME, and run "rpm -ta courier-imap-X.tar.gz" where X is the release number
  5. Once rpm is done compiling, packages are located in $HOME/rpm/RPMS/i386. You just need to install courier-imap-X.i386.rpm, but need the other packages for extra features (user authentication against an LDAP, MySQL, or PostgreSQL server). Log on as root, cd to the user's home directory above, and run "rpm -Uvh rpm/RPMS/i386/courier-imap-X.i386.rpm" . Most of the stuff lives in /usr/lib/courier-imap/ .
  6. Launch Courier with /etc/rc.d/init.d/courier-imap start. As this is the first time Courier is running, it will start by creating SSL certificates
  7. Create a user account in /etc/passwd or /etc/shadow through "useradd jdoe ; passwd jdoe"
  8. Run "maildirmake /home/jdoe/Maildir", followed by "chown -R jdoe.jdoe /home/jdoe/Maildir" to create a maildir directory structure
  9. Run "telnet localhost pop3", followed by user jdoe pass mypassword to check that Courier-IMAP authenticates the user correctly
  10. Run "telnet localhost imap" to check that Courier is listening to the IMAP port, and exit with ".1 logout"

Resources

Windows IMAP clients

I focus exclusively on Windows clients because few corporate users work on Linux clients, while the "server = Linux, clients = Windows" setup is very common. As explained above, I have yet to find an IMAP client that supports automatic synchronization between client and server: For instance, Eudora doesn't copy any sent e-mails from your local Out mailbox to your remote mailbox on the IMAP server (you must copy/paste them yourself), Netscape displays a bogus Local tree, when in fact you only need the remote tree at all times (if you have switched to "offline" mode, this remote tree is really a local representation of your remote mailboxes. When you switch back to online mode, all operations done to this local cache are played back on the remote server). Outlook Express also requires you to copy/paste messages to keep client and server in sync. <Sigh>

Netscape

As I couldn't find documentation about Netscape's support for disconnected mode, it took me a while to figure out that disconnected mode does not mean that you need to map folders on the IMAP server and on your local host. Instead, you must first configure Netscape to tell it which remote folders are affected when you go offline (so Netscape knows which folders should have a local copy when you go "offline").

Once this is done, all you need is to use File | Go Offline: From now on, what is shown as the remote folders (eg. the "imap.acme.com" tree) are really local copies on your computer, and any operation made while you are in offline mode is only made on this local representation. Operations will be replayed on the IMAP server only once you have switched back to online mode (ie. e-mails that you wrote while offline will be sent, new folders will be created, folders will be renamed if you did so locally, etc.)

In other words, the Local Mail tree is very misleading. It has nothing to do with IMAP. In IMAP mode, it's only used as a buffer for sent/draft e-mails, etc. Remember that you must subscribe to remote mail folders for them to show up in the tree list (File | Subscribe...).

Also, I couldn't copy/paste messages from my local folders to remote folders other than the Inbox, eg. copy from Local | Sent to Remote | Outbox. The message is there (cat /home/jdoe/mail/Outbox), but Netscape won't display them.

To delete e-mails in your remote mailboxes, select them and hit DEL, followed by File | Compress this folder.

Check how to have sent e-mails be uploaded to the remote Outbox when going back online

Eudora

Eudora has been my favorite e-mail client for years, but fact is, its support of IMAP isn't the best. Basically, it doesn't support disconnected mode: Sent e-mails are only saved in the local Out mailbox, and you are required to copy/paste them manually to your Out mailbox on the IMAP server. Likewise, the Offline mode really means that Eudora will not send any e-mail while you have set it to Offline mode (those e-mails will be marqued as Queued).

When the Eudora documentation talks about synchronizing mailboxes, they actually mean refreshing the view of the mailboxes in ~/Mail in case other people were working on them from other computers. It does not mean that the local mailboxes sitting on the client host will be sync with whatever is available in the ~/Mail mailboxes.

To check for any new e-mail in your remote INBOX, right-click on this remote folder, and select Resynchronize Mailbox.

Deleting e-mails in your remote mailboxes requires two steps: Select the e-mails and press DEL to mark them as Deleted, followed by Message | Purge Messages. This is actually a requirement of the IMAP protocol. If you want files to be deleted/moved in one go, Tools | Options | Incoming Mail | When I Delete a Message = Move it to Trash. Messages will be moved accordingly, but a copy is saved in your local Trash mailbox. To tell Eudora to empty it automatically when exiting, Tools | Options | Miscellaneous | Empty Trash when exiting

To create an IMAP account (ie. another "persona" in addition to the existing POP3, local account):

Fill in the Generic Properties tab:

Fill in the Incoming Mail tab:

Note: When using Cyrus IMAP, you can leave the Mailbox Location empty as all transactions go through the IMAP protocol (ie. the IMAP client will not try to connect to your home directory on the Linux server; CHECK IF REALLY NEED TO USE ~/mail WITH UoW). As for the Authentication style section, Passwords and CRAM-MD5 might have you believe that this reflects the different between password hashing as done in /etc/passwd (Crypt) and /etc/shadow (MD5), but it makes no difference with our setup.

Once this is done, a new top-level mailfolder appears in Eudora's mailfolder window (here, IMAP). For the time being, it only contains one mailbox, Inbox, which really is a pointer to /var/spool/imap/user/user/jdoe/ (Cyrus), /var/spool/mail/jdoe (UoW), or /home/jdoe/mbox (UoW if using an mbox file in users' home directory.)

To keep local copies of e-mails located on the remote folders, just drag-and-drop them while keeping the SHIFT key down (otherwise, e-mails will be moved to the local folder):

To go offline, right-click on the left-most square in Eudora's task bar, and select (duh) Go Offline:

In case other people share some of you mailboxes on the IMAP server, have Eudora refresh the list by right-clicking on the IMAP server and select Refresh Mailbox List:

Rants

  1. Create an IMAP account, and send an e-mail: It is saved in the (local) Eudora tree. If you want to keep a copy of this e-mail that you sent, you'll have to copy/paste this e-mail to your remote Out folder on the IMAP server
  2. Why does Eudora display a list of any file listed in the user's home directory, eg. .bashrc ?

Outlook Express

How to use OE to work with an IMAP server

(Updated Feb 08, 2002) In a nutshell, as of today, OExpress is the only IMAP client for Windows that I know that supports disconnected mode, ie. any action that you perform on the folders and e-mails that are located on your client computer are replayed on the IMAP server once you're back online. The way it works, is that you select which folders you wish to replicate on your client computer, switch to offline mode through File | Work Offline, perform any task while offline, and go back online by unchecking Work Offline: Voilą! OExpress will now replay all the tasks that you performed offline.

  1. Create an account to the IMAP server by selecting Tools | Accounts, click on Add | Mail...
  2. Click Yes when prompted to download the list of folders from the IMAP server. OExpress ends up listing three folders (Inbox : /var/spool/mail/jdoe, while Sent Items and Drafts are files that OExpress creates in the user's home directory), along with whatever files and sub-directories happen to exist in the user's home directory)
  3. Send an e-mail to the user account to which you connect through OExpress by running:

    mail -s "Testing IMAP" jdoe
    This is a test. This message should show up in OExpress.
    .
  4. Hit Tools | Send & Receive | Receive All, and check that this test e-mail shows up in the Inbox folder of mail.acme.com (where mail.acme.com is the name of the remote IMAP server)
  5. To access your folders while working in disconnected mode, select the IMAP mail account in the mail tree in the left hand-side, and select the ad hoc folders. This tells OExpress which folders should be saved locally and made accessible while disconnected from the server.

Rants

  1. The presence of a "Local Folders" section doesn't make sense when using OE to work with an IMAP server...
  2. You cannot create folders while in disconnected mode!
  3. Following the IMAP protocol, any deleted/moved e-mail must be purged, ie. it takes two steps before the e-mail is actually deleted/moved on the server (Edit | Purge Deleted Messages.)
  4. After you created a new folder on the server, remember to check it in the Synchronization Settings column, or e-mails won't be copied locally
  5. Remember to always synchronize your laptop with the IMAP server before going offline. Otherwise, when you move an e-mail from one folder to another (eg. Inbox to Old In), the e-mail will not be accessible inside the target folder. In other words, while online, if you just move an e-mail, go offline, and try to view the e-mail, OExpress will prompt you with "You are currently working offline. Would you like to go online now?", and display "Message unavailable while offline" if you answer no...
  6. While offline, if you move or delete an e-mail, you'll have to hit Edit | Purge Deleted Messages once you're back online: OE doesn't remember to perform this task for you...
  7. By default, OE creates two folders in the user's home directory, Drafts and Sent Items. I couldn't remove those two from OE, and had to delete those two files by SSHing to the IMAP server... Theres is an option to tell OE to ignore those two folders (select the IMAP profile's properties, select the IMAP tab, and uncheck "Store special folders on IMAP server"), but like it or not, it will create them when you create a new IMAP profile...
  8. If you'd rather use an Old Out folder on the IMAP server instead of the Sent Items folder that OE creates, e-mails that you sent are not synced from Local Folders | Sent Items to mail.acme.com | Old Out....

The Bat

Rants

  1. When creating a new e-mail account, The Bat displays only "POP3 Server" instead of letting the user choose the IMAP protocol right away instead of having to complete this part and go to the Properties | Transport section, and change it to IMAP4.
  2. Even if you set the account to use IMAP... it uses POP3, ie. deletes e-mails from the server!
  3. Uses SDI windows to write/display a message instead of MDI windows like Eudora

Cyrusoft Mulberry

Rants

This is supposed to be the only IMAP client for Windows that really supports disconnected mode, but it's too bad the UI... isn't for everyone

Outlook

Offline folders are stored in the offline folder file (OST). This file is located on your computer hard disk and is available even when the network is down. The OST can be compacted to save space on your computer. Offline folders are only available with Microsoft Exchange Server.

Pegasus

At first sight, doesn't seem to support disconnected mode.

Q&A

(Outlook Express) When checking my e-mails on a remote IMAP server, there's a delay of several seconds

Check that you are not running a firewall on the client host, in which case this delay is due to the fact that the server is trying to connect to Identd (port 113), and times out since the firewall doesn't answer. Another possibility for this delay is that your client computer is also connected to the Net through eg. ADSL: To tell OEexpress to connect through the LAN, select Tools | Accounts | Mail, select the connection profile used to connect to the remote IMAP server, select Properties | Properties, and choose "Always connect to this account using" = LAN.

What are IMSP and ACAP?

Internet Message Support Protocol (IMSP) is the older of the two, and is designed to store IMAP client configuration and personal address books. Application Configuration Access Protocol (ACAP) handles storage of Internet application data and configuration, including address books, bookmarks for web browsers, etc.

What is SASL?

It is a library that provides standard authentication for any Internet application.

What is SIEVE?

A server-side filtering language, more basic than Procmail

What is CRAM?

Challenge-Response Authentication Mechanism, a carry-over from the Andrew Messaging System (AMS.) It provides encrypted authentication without requiring Kerberos. The latest versions of Cyrus support SASL instead. Encrypted authentication methods include CRAM-MD5, DIGEST-MD5, and GSSAPI (MIT Kerberos 5.)

Note that "Unix authentication" means that the password is sent over the wire in clear text.

How does Cyrus IMAP keep track of mailboxes and their ACL?

(CHECK) /etc/imap/mailboxes.

Does Cyrus IMAP need Unix user accounts?

Not if you use CRAM-MD5 and /etc/sasldb. O'Reilly's Managing IMAP mentions a utility called pwcheck that is supposed to be used when using /etc/shadow instead of /etc/passwd because Cyrus cannot access /etc/shadow due to its running as cyrus:mail, but it was nowhere to be found in the RPM's used in this tutorial. I guess PAM solves the issue.

What is LMTP?

Local Mail Transport Protocol allows an SMTP server to deliver mail to an IMAP/POP server, and receive an ACK about delivery. The two servers do not have to be located on the same host.

What is new in Cyrus IMAP 2.0?

Mail formats

Standard Unix mail is known as mbox or Berkeley format, and all messages for a given folder is saved in one file. Cyrus, on the other hand, saves messages in a format similar to MH format, where each message is stored in an individual file. Each file has a format like Berkeley mail, but it is terminated with a CRLF character. Messages are saved in independent directories below the user's top-level directory (eg. /var/spool/imap/user.jdoe/). Directories are called "mailboxes" in IMAP parlance. In addition to individual messages, each mailbox contains system files (cyrus.index, cyrus.header, cyrus.cache, and cyrus.seen).

What are namespaces user for in Cyrus IMAP?

When building and installing the Courier IMAP RPM, how do I know which authentication modules it supports?

Resources

http://www.e-trend.de/~toens/ You could use a SSL wrapper to secure the connections. There are som documents available on the web, at leas http://security.fi.infn.it/tools/stunnel/index-en.htm could be helpful to you. There's also a stunnel FAQ at http://www.stunnel.org/faq/ and a SSL tunnel & IMAP / Pop / SMTP document at http://www.octaldream.com/scottm/talks/ssl/stunnel.html