My Email Setup with Notmuch and Alot

Email is annoying. Doubly so when you’re a contractor and have a separate inbox for every client. Having to keep 4 Gmail tabs open to keep track of incoming requests feels like a waste of brain power and space.

I’ve spent a long time looking for a solution that solves all my needs:

• Gives me a unified inbox across all my email accounts on my Linux desktop
• Lets me receive notifications for important emails anywhere (desktop, laptop or phone)
• Allows me to quickly scan and archive emails that I need to see, but don’t need to action (notifications, alerts, etc.)
• Supports authentication with Gmail accounts via OAuth

The first and most obvious solution is a GUI mail client. Mailspring, Thunderbird, Evolution and many of others have crossed my desktop. These are all great tools, but ultimately I find that:

• Configuring them with Gmail OAuth is awkward or not possible
• They are slow, especially with a lot of emails cached locally
• Search performance isn’t great - I end up having to login to Gmail whenever I need to search

I’m not the sort of person who lives in the terminal, I like a good GUI when it serves a purpose and having previously ventured in to to trying mutt and not wanting to invest the time to learn the keyboard shortcuts, I wasn’t particularly inclined to go down that route again. How naive I was.

Some time ago I had configured isync to keep local backups of my mail. If all my mail was being regularly synced for backups in maildir format, couldn’t I read it directly from there?

• Thunderbird has Maildir support but has some scary warnings about it being an experimental feature.
• mutt/neomutt are nice, but the aforementioned fear of learning a bunch of new keybindings kept me away - it’s also a bit of a pain to configure with multiple inboxes and search is still slow.

Eventually, I rediscovered notmuch and a well-supported client, alot - which I’ve been using for a few months very happily. Let’s tackle the setup:

Step 1: Grabbing email

The first thing I need is to grab all my email from my various mail providers and copy those emails locally. This gives me local backups, the ability to search (I’ll come back to that in a bit), and a way to quickly access, read and manipulate email locally.

To do this, I use isync (the tool is called isync, the binary is called mbsync) to synchronise a local copy with each IMAP server. My config looks a bit like this:

~/.config/mbsync/mbsyncrc

Expunge None
Create Near
SyncState *

IMAPAccount my-email@gmail.com
Host imap.gmail.com
User my-email@gmail.com
AuthMechs XOAUTH2
PassCmd "oauth2get google my-email@gmail.com"
SSLType IMAPS
SSLVersions TLSv1.1 TLSv1.2

IMAPStore my-email@gmail.com-remote
Account my-email@gmail.com

MaildirStore my-email@gmail.com-local
Path ~/mail/my-email@gmail.com/
Inbox ~/mail/my-email@gmail.com/Inbox
SubFolders Verbatim

MaildirStore my-email@gmail.com-archive
Path ~/mail/archive-my-email@gmail.com/

Channel my-email@gmail.com-archive
Far ":my-email@gmail.com-remote:[Google Mail]/All Mail"
Near ":my-email@gmail.com-archive:Archive"
Sync All

Channel my-email@gmail.com-trash
Far ":my-email@gmail.com-remote:[Google Mail]/Bin"
Near ":my-email@gmail.com-archive:Trash"
Sync Pull

Channel my-email@gmail.com-drafts
Far ":my-email@gmail.com-remote:[Google Mail]/Drafts"
Near ":my-email@gmail.com-local:Drafts"
Sync Pull
Expunge Both

Channel my-email@gmail.com-sent
Far ":my-email@gmail.com-remote:[Google Mail]/Sent Mail"
Near ":my-email@gmail.com-local:Sent"
Sync Pull
Expunge Both

Channel my-email@gmail.com-inbox
Far ":my-email@gmail.com-remote:INBOX"
Near ":my-email@gmail.com-local:Inbox"
Sync All
Expunge Both

Group my-email@gmail.com
Channel my-email@gmail.com-trash
Channel my-email@gmail.com-inbox
Channel my-email@gmail.com-drafts
Channel my-email@gmail.com-sent
Channel my-email@gmail.com-archive

# ... repeat for every mail account

Group all
Channel my-email@gmail.com-trash
Channel my-email@gmail.com-inbox
Channel my-email@gmail.com-drafts
Channel my-email@gmail.com-sent
Channel my-email@gmail.com-archive
# ... repeat for every mail account

For each of my email accounts, this defines:

1. An IMAPAccount containing the server details and credentials for the account.
2. An IMAPStore for that account. This tool syncs between two stores - so we define this to give it an entity from which to sync.
3. Two MaildirStore entries - one for ‘active’ email, and one for the archived email - keeping them separate makes it easier to backup only the archives and gives me the option to access only the ‘active’ emails if I want to use clients like mutt.
4. A set of Channels - the config entry used for defining how two stores are synchronised. I create a seperate channel for each key folder within my remote IMAP store so that I can keep them consistently named locally across different mail providers.
5. A Group listing all my defined channels, so that running mbsync all syncs everything.

I have mbsync all configured to run on a systemd timer every 2 minutes:

~/.config/systemd/user/mbsync-run.service

[Unit]
Description=Sync mail with mbsync
OnFailure=status-email-user@%n.service

[Service]
Type=oneshot
Nice=10
ExecStart=/usr/bin/mbsync --config /home/tom/.config/mbsync/mbsyncrc all
Environment="HOME=/home/tom"

~/.config/systemd/user/mbsync-run.timer

[Unit]
Description=Sync emails with mbsync on schedule

[Timer]
OnBootSec=2m
OnUnitActiveSec=2m
Persistent=true

[Install]
WantedBy=timers.target

Step 2: Indexing

Now all my emails are being synced locally on a schedule, I need some way to make sense of them. This is where notmuch comes in.

notmuch scans your maildir formatted directory and adds everything in to a Xapian index giving you a nice CLI tool to search through all your emails really quickly.

Running notmuch new will walk you through getting it all set up - you can then edit your config file as you need.

export NOTMUCH_CONFIG=$XDG_CONFIG_HOME/notmuch/config

~/.config/notmuch/config

[database]
path=/home/tom/mail

[user]
name=Tom Usher
primary_email=tom@tomusher.com
other_email=my-email@gmail.com

[new]
tags=new;
ignore=

[search]
exclude_tags=deleted;spam;

[maildir]
synchronize_flags=true

The most important thing here for me is the tags definition in the [new] section - this tells Notmuch that when it finds new mail, it should mark it in its internal tagging system as new. The default option here is to set both the new and unread tags, but as I may have read this email using a different client, I prefer the unread flag to by synchronised from the maildir using the synchronize_flags setting.

To tell notmuch to index new mail when it comes in, I add an ExecStartPost command to my systemd unit:

~/.config/systemd/user/mbsync-run.service

[Unit]
Description=Sync mail with mbsync
OnFailure=status-email-user@%n.service

[Service]
Type=oneshot
Nice=10
ExecStart=/usr/bin/mbsync --config /home/tom/.config/mbsync/mbsyncrc all
ExecStartPost=/usr/bin/notmuch new
Environment="HOME=/home/tom"

Now I can do all sorts of handy things with the notmuch CLI:

• notmuch search tag:unread - list all the emails I haven’t read
• notmuch tag -unread -- tag:unread - remove the unread tag from all the mail tagged unread.
• notmuch tag important -- tag:unread and critical - tag anything unread containing the term ‘critical’ as important.
• notmuch search date:2020-12-01..2020-12-31 and from:auto-confirm@amazon.co.uk - find all my order receipts from Amazon in December 2020.

This is useful, but it’s not the nicest way to regularly read email. That’s where the number of available notmuch frontends comes in.

3. Reading Email

Of the frontend applications supporting notmuch that I’ve tried, I ended up sticking with alot which I find to be a great balance between simplicity and customisability.

alot has a bunch of neat features, but it does what I want particularly well:

• A simple overview of mail across all my inboxes in one panel
• Very quick to start and flick through emails
• Lets me archive emails quickly with the default a keybind - which toggles the inbox flag

I don’t customise it much, other than defining an ‘account’ for sending email, and adding a useful open in browser custom hook - which lets me press a key to view an email in the browser - useful for those complicated HTML emails.

[accounts]
    [[personal]]
        realname = Tom Usher
        address = tom@tomusher.com
        sendmail_command = msmtp --account=personal -t
        sent_box = maildir:///home/tom/mail/my-email@gmail.com/Sent
        draft_box = maildir:///home/tom/mail/my-email@gmail.com/Drafts

[bindings]
  [[thread]]
  b = call hooks.open_in_browser(ui)

4. Sending email

I don’t actually do much email sending - for better or for worse most of my communication has moved to messaging platforms.

It is handy to have something to easily send emails without launching Gmail, and for that I use msmtp. This can be configured using a relatively straightforward config file:

~/.config/msmtp/config

defaults
auth    on
tls     on
tls_trust_file  /etc/ssl/certs/ca-certificates.crt
host        smtp.gmail.com
port        587
auth        oauthbearer

# Personal
account     personal
from        tom@tomusher.com
user        my-email@gmail.com
passwordeval    oauth2get google my-email@gmail.com

# ... repeat for each account

account default : personal

I can then use the ‘account’ I previously configured in alot to send and reply to emails.

5. Making it work with Gmail

Most clients I work with use GSuite for email - it’s quick to set up, works well and provides an excellent web-based client.

Because I’ve gone down this whole weird route, I do need to make some adjustments to cope with Gmail’s oddities.

First up; ‘archiving’ an email via IMAP in Gmail might not be what you expect. Incoming emails end up in both your All Mail folder as well as your Inbox. To archive something, you need to delete it from your Inbox, leaving the copy in All Mail intact.

I have a script which does that on the local copy of my inboxes by searching for files in any Inbox folder that no longer have the tag inbox (i.e. I’ve archived them in alot), use grep to filter the results to only show mail from my Inbox (as notmuch search will return every copy of the email it found, including the one in my Archive folder), then delete the remaining items.

Once they’ve been removed, mbsync will synchronise those changes back to Gmail, removing the copy from my Gmail Inbox, effectively archiving the email.

~/.mail/scripts/notmuch-move.sh

#!/bin/bash
echo "Removing emails deleted from inbox"
notmuch new --no-hooks
notmuch search --output=files --format=text folder:/Inbox/ -tag:inbox | grep Inbox | xargs --no-run-if-empty rm
notmuch new --no-hooks

The next problem is that if I have the Gmail web client open, or am checking emails on my phone, I’m likely to have read and archived an email before it hits my computer. All new incoming email will be marked as new by notmuch, even if it has already been read or archived.

To resolve that, I treat the new tag as ‘to be processed’, and do all my post-processing on emails with that tag. For example, this script tags everything new and in an Inbox folder with the inbox tag, and then removes the new tag from everything:

~/.mail/scripts/notmuch-tag.sh

#!/bin/bash
echo "Tagging new inboxed mail"
notmuch tag +inbox -- folder:/Inbox/ and tag:new
notmuch tag -new -- tag:new

Both these run as part of my systemd unit file:

[Unit]
Description=Sync mail with mbsync
OnFailure=status-email-user@%n.service

[Service]
Type=oneshot
Nice=10
ExecStartPre=/home/tom/mail/scripts/notmuch-move.sh
ExecStart=/usr/bin/mbsync --config /home/tom/.config/mbsync/mbsyncrc all
ExecStartPost=/usr/bin/notmuch new
ExecStartPost=/home/tom/mail/scripts/notmuch-tag.sh
Environment="HOME=/home/tom"

Finally, there’s the problem of authorization. For Google accounts, I prefer to authorize with OAuth so that I can make use of 2FA and avoid creating insecure app passwords. To do this, I use the oauth2token project, which provides two binaries to create and get an OAuth token from local storage. mbsync and msmtp are then configured using PassCmd and passwordeval respectively to fetch the token using oauth2get, which will automatically refresh the token when required.

The XOAUTH2 mechanism requires an implementation of SASL with a XOAUTH2 plugin installed; I use cyrus-sasl and cyrus-sasl-xoauth2.

This also requires creating an OAuth2 credential in Google Cloud Console - which is a bit outside the scope of this post, but be aware you’ll need to add the https://mail.google.com/ scope to your application, which is a ‘restricted scope’ and makes it extra awkward to do things:

• You can keep the application in testing mode, but your auth tokens will expire after 7 days, so you’ll need to re-authorize annoyingly frequently.
• In testing mode, you’ll get an error if you try to authorize and are logged in with more than one Google account.

To fix those issues, you can choose to ‘Publish` the application, which will prompt for manual verification by Google, but you don’t actually need to do this - you’ll only get a scary warning whenever you try to login.

6. Extra Fun Things

Now you have a nice CLI for querying your mail, you can make it work for you.

For example, in my personal notmuch-tag script, I add the notify tag to every email that’s unread, new and in my inbox. I have the following script running in an ExecStartPost on my systemd unit which notifies me of these new emails, and then removes the notify tag so I don’t get duplicate notifications:

#!/bin/bash
SEARCH="tag:notify"

NOTIFY_COUNT=$(notmuch count "$SEARCH")

if [ "$NOTIFY_COUNT" -gt 0 ]; then
  RESULTS=$(notmuch search --format=json --output=summary --limit=3 --sort="newest-first" "$SEARCH" | jq -r '.[] | "\(.authors): \(.subject)"')
  notify-send "$NOTIFY_COUNT New Emails:" "$RESULTS"
fi

notmuch tag -notify -- tag:notify

Another handy tool is my count of new emails in my polybar:

~/.config/polybar/scripts/mail.sh

#!/bin/bash

EMPTY_INBOX_ICON="%{T3}%{T-}"
UNREAD_INBOX_ICON="%{T2}%{T-}"

UNREAD=$(notmuch count is:inbox and is:unread)
if [[ $1 = "count" ]]; then
   if [ $UNREAD = "0" ]; then
       echo $EMPTY_INBOX_ICON $UNREAD
   else
       echo $UNREAD_INBOX_ICON $UNREAD
   fi
elif [[ $1 = "alot" ]]; then
   $TERMINAL -e alot &
fi

~/.config/polybar/config

# ...

[module/mailcount]
type = custom/script
interval = 1
format-padding = 0
format = <label>

exec = ~/.config/polybar/scripts/mail.sh count
click-left = ~/.config/polybar/scripts/mail.sh alot

# ...

Summary

While this was a fun thing to set up and a great way to play with some more open source goodies, I’ve found it works well for me for a number of reasons:

• Being able to quickly scan and archive all my emails across 4+ email accounts every morning has absolutely saved me time and frustration.
• I still get all the benefit of being able to set up mobile email clients with IMAP and everything syncs as you’d expect.
• When I’m actually sitting at my desk, this approach lets me deal with emails quickly with minimal distractions without having to flick through various tabs of different webmail providers.
• Even though I’m still a little tied to Gmail, it feels liberating to have all my emails locally, backed up and safely stored.