Le weblog entièrement nu

Roland, entièrement nu... de temps en temps.

Gnus, Dovecot, OfflineIMAP, search: a HOWTO

A long time ago, when I was first introduced to email, I was using the Mail program from Unix. I quickly converted to Elm, then Mutt, which were both better in terms of interface. Then I found out about Gnus, and I wouldn't dream of letting it go now. However, Gnus has started showing its age several times, and several times have I needed to upgrade the way I was using it: first because I needed to sort and split email, then because I took the sorting out of Gnus and into Procmail for more advanced filtering (including spam filtering), then because I switched to storing the emails on an IMAP server so I could read them remotely from several computers. My setup as of a few days ago was functional, but since I have grown over time to splitting emails into several hundred folders, checking for new messages was becoming more and more boring.

So it's time to jump in with all the cool kids and switch to a modern solution: still Gnus of course, but with Dovecot, OfflineIMAP for synchronisation, and let's add email searches into the mix while we're at it. My web searches didn't turn up a simple step-by-step HOWTO, but I assembled bits from different places, and here's my attempt at documenting my new setup.



Dovecot setup

We'll use Dovecot as a local IMAP server. And since we're lazy, we'll access it over a pipe, and dispense with the network part.

OfflineIMAP setup

OfflineIMAP is basically an optimised two-way synchronisation mechanism between two email “repositories”. We'll use it in IMAP-to-IMAP mode.

 accounts = MyAccount
 pythonfile = .offlineimap.py

 [Account MyAccount]
 localrepository = LocalIMAP
 remoterepository = RemoteIMAP
 # autorefresh = 5
 # postsynchook = notmuch new

 [Repository LocalIMAP]
 type = IMAP
 preauthtunnel = MAIL=maildir:$HOME/Maildir /usr/lib/dovecot/imap
 holdconnectionopen = yes

 [Repository RemoteIMAP]
 type = IMAP
 remotehost = mail.example.com
 remoteuser = jsmith
 remotepass = swordfish
 ssl = yes
 nametrans = lambda name: re.sub('^INBOX.', '', name)
 # folderfilter = lambda name: name in [ 'INBOX.important', 'INBOX.work' ]
 # folderfilter = lambda name: not (name in [ 'INBOX.spam', 'INBOX.commits' ])
 # holdconnectionopen = yes
 maxconnections = 3
 # foldersort = lld_cmp
 # Propagate gnus-expire flag
 from offlineimap import imaputil

 def lld_flagsimap2maildir(flagstring):
     flagmap = {'\\seen': 'S',
                '\\answered': 'R',
                '\\flagged': 'F',
                '\\deleted': 'T',
                '\\draft': 'D',
                'gnus-expire': 'E'}
     retval = []
     imapflaglist = [x.lower() for x in flagstring[1:-1].split()]
     for imapflag in imapflaglist:
         if flagmap.has_key(imapflag):
     return retval

 def lld_flagsmaildir2imap(list):
     flagmap = {'S': '\\Seen',
                'R': '\\Answered',
                'F': '\\Flagged',
                'T': '\\Deleted',
                'D': '\\Draft',
                'E': 'gnus-expire'}
     retval = []
     for mdflag in list:
         if flagmap.has_key(mdflag):
     return '(' + ' '.join(retval) + ')'

 imaputil.flagsmaildir2imap = lld_flagsmaildir2imap
 imaputil.flagsimap2maildir = lld_flagsimap2maildir

 # Grab some folders first, and archives later
 high = ['^important$', '^work$']
 low = ['^archives', '^spam$']
 import re

 def lld_cmp(x, y):
     for r in high:
         xm = re.search (r, x)
         ym = re.search (r, y)
         if xm and ym:
             return cmp(x, y)
         elif xm:
             return -1
         elif ym:
             return +1
     for r in low:
         xm = re.search (r, x)
         ym = re.search (r, y)
         if xm and ym:
             return cmp(x, y)
         elif xm:
             return +1
         elif ym:
             return -1
     return cmp(x, y)

The first part of this file adds a new flag that OfflineIMAP will propagate back and forth. By default, only the standard IMAP flags are propagated; we also want to synchronize the gnus-expire flag that Gnus uses to mark expirable articles. It's a hack, but it works for now (maybe someday OfflineIMAP will propagate all the flags it finds?).

The second part of that file can be dispensed with (and won't be used unless the foldersort option is uncommented in .offlineimaprc): it's only there to ensure that some important folders are propagated first, and some others go last. I don't know exactly how they are sorted by default, but I'd like the most important ones to come first, so I can start reading them while the archives and the spam are still being fetched.

Gnus configuration

(require 'offlineimap)
(add-hook 'gnus-before-startup-hook 'offlineimap)

Bonus: email searches

The simple way:

(require 'nnir)

This goes in your .gnus. Then your group buffer will get a new command. Mark some folders with #, then M-x gnus-group-make-nnir-group (or use the G G shortcut), and type in a set of keywords. This search is performed by the IMAP server (Dovecot), which may or may not be very efficient, especially if you select many folders.

Bonus+: email searches, faster

The real cool kids use Notmuch nowadays, at least for email indexing and searching. It's fast, it allows complex queries, and it's generally cool. The downside is that it uses up quite some disk space for its indices, in addition to the actual emails. For that reason I'll keep it to my main computer, and I'll stick to nnir on my laptop (which has the same setup apart from that).

(require 'notmuch)
(add-hook 'gnus-group-mode-hook 'lld-notmuch-shortcut)
(require 'org-gnus)

(defun lld-notmuch-shortcut ()
  (define-key gnus-group-mode-map "GG" 'notmuch-search)

(defun lld-notmuch-file-to-group (file)
  "Calculate the Gnus group name from the given file name.
  (let ((group (file-name-directory (directory-file-name (file-name-directory file)))))
    (setq group (replace-regexp-in-string ".*/Maildir/" "nnimap+local:" group))
    (setq group (replace-regexp-in-string "/$" "" group))
    (if (string-match ":$" group)
        (concat group "INBOX")
      (replace-regexp-in-string ":\\." ":" group))))

(defun lld-notmuch-goto-message-in-gnus ()
  "Open a summary buffer containing the current notmuch
  (let ((group (lld-notmuch-file-to-group (notmuch-show-get-filename)))
        (message-id (replace-regexp-in-string
                     "^id:" "" (notmuch-show-get-message-id))))
    (setq message-id (replace-regexp-in-string "\"" "" message-id))
    (if (and group message-id)
    (switch-to-buffer "*Group*")
    (org-gnus-follow-link group message-id))
      (message "Couldn't get relevant infos for switching to Gnus."))))

(define-key notmuch-show-mode-map (kbd "C-c C-c") 'lld-notmuch-goto-message-in-gnus)


This setup can be replicated on several computers, of course. I have it on two, and there's no reason I couldn't have more. The flags do get propagated back and forth, including the Gnus-specific “expirable” flag. Accessing the local Dovecot is much faster than going through the DSL to the “master” IMAP server, and I'm pretty convinced that OfflineIMAP and its multi-threading is also faster than Gnus is, even talking to the same remote server. The email searching with Notmuch is a nice bonus, especially since they're fast too (and this despite my 8-year-old computer).

There are a few minor glitches. I can live with them, but I should let you know anyway.

Apart from that, I'm pretty happy with this new setup. So I hope this documentation will be useful to others, so I can spread the happiness around. Send your thanks to the authors of the software involved (Gnus, Dovecot, OfflineIMAP, offlineimap.el, Procmail, and so on).

Let's see how many years I'll keep that system!

Update: I'm told that starting with version 2.0 of Dovecot, the correct way to tell the server where to find the mail is something like /usr/lib/dovecot/imap -o mail_location=maildir:$HOME/Mail rather than MAIL=maildir:$HOME/Maildir /usr/lib/dovecot/imap. If you're using that version, you probably need to make the change (both in OfflineIMAP's and Gnus's configuration).

Creative Commons License Sauf indication contraire, le contenu de ce site est mis à disposition sous un contrat Creative Commons.