Develop website message function based on IMAP mailbox

The site has been online for a few months now and there has been no message function. A friend suggested adding a message or discussion support. My earliest thought was that leaving a message is a very low frequency operation, and I also left my email address on my personal page. If a friend really needs to discuss a problem, he or she can send an email. But the actual situation is that using email is relatively rare and inconvenient. The common disadvantage of Disqus and Github is that they require registration and have certain privacy risks. I wanted to find a system that supports anonymous comments, and finally found isso. isso uses sqlite to save comment data, supports anonymous comments, and fits my needs in every way. However, it was developed in python, which made it very cumbersome to deploy. Finally, I came up with a rather tricky solution: a simple message board based on IMAP mailbox. Today, I’ll put together the ideas and solutions and share them with you.

The comment or message system is also very complex to make complete. Like tree structure, building in building, multi-level replies, likes, etc. But as a blog comment system, is it really necessary to do so complicated? I think not. The core function of a blog commenting system is to provide a communication channel for readers and authors, to establish a connection between readers and authors. Therefore, how to display comments, whether to support quotes and replies are in the secondary. The core function is that readers’ messages can be received by authors as soon as possible, and authors can easily reply to readers’ questions. From this point of view, there is no big problem in using email to communicate. But, after all, email is not commonly used in China, so we need to find ways to lower the threshold of using messages. For this reason, I hit on the idea of IMAP in the mail system.

Those who know about email must have heard of the terms SMTP/POP/IMAP. Simply speaking, SMTP is used to send emails and POP/IMAP to view emails. The emails sent by others to your email address are saved on the server, and you can configure the client of your cell phone or computer to view the contents of emails. POP protocol can only download emails from the server, while IMAP protocol is more complicated, not only can download, but also upload! What 🤔 can be uploaded? The Apple system’s notes software supports saving notes to the mail server, using the IMAP upload function. Then can we upload users’ messages to mailboxes via IMAP as well?

So I started to study the IMAP protocol. It’s a complex protocol, to be fair. Instead of looking into the protocol itself, I tried searching for a Go language IMAP protocol library. I found go-imap. go-imap’s Client object supports uploading emails to an IMAP server via the Append() method to upload emails to the IMAP server. After uploading, the effect is equivalent to receiving a new email and the client will be alerted. Example code is directly attached to the documentation.

package main

import "github.com/emersion/go-imap/client"

func main() {
  // 登录 IMAP 服务器
	c, err := client.DialTLS("mail.example.org:993", nil)
	if err != nil { panic(err) }
	defer c.Logout()
	err := c.Login("username", "password")
  if err != nil { panic(err) }
  // 构造邮件内容
  var b bytes.Buffer
  b.WriteString("From: <root@nsa.gov>\r\n")
  b.WriteString("To: <root@gchq.gov.uk>\r\n")
  b.WriteString("Subject: Hey there\r\n")
  b.WriteString("\r\n")
  b.WriteString("Hey <3")
  // 上传邮件到收件箱 INBOX
  err := c.Append("INBOX", nil, time.Now(), &b)
}

The whole process is relatively simple and will not be repeated. However, the second step of constructing the content of the email needs to be modified. The email content in the example is in English.

From: <root@nsa.gov>\r\n
To: <root@gchq.gov.uk>\r\n
Subject: Hey there\r\n
\r\n
Hey <3

Where From, To and Subject are all specified, corresponding to sender, recipient and email subject. Because the email protocol was invented by Americans, it did not support Chinese in the early days, and even many western European characters were not supported. In order to solve this problem, people introduced MIME protocol, which is called Multipurpose Internet Mail Extensions, a kind of protocol provision for extending emails, through which we can add HTML tables, pictures, Chinese and even file attachments to emails. The following is a MIME message with Chinese characters.

Mime-Version: 1.0
From: =?utf-8?q?=E5=BC=A0=E4=B8=89?= <z3@foo.com>
Subject: =?utf-8?b?5YWz5LqO5rab5Y+U?=
To: <hi@taoshu.in>
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
Date: Fri, 05 Nov 2021 05:52:37 +0000

=E4=BD=A0=E5=A5=BD

https://taoshu.in/about.html

Notice that there is an additional Mime-Version header to specify the MIME protocol version. With this field, it means this is a MIME message.

The following From field seems to be garbled

`1`	`=?utf-8?q?=E5=BC=A0=E4=B8=89?=`

Don’t be afraid. It begins and ends with =? and ? =, you can ignore them. The first question mark followed by utf-8 means that the content after it is in utf-8 encoding. The ?q? after the first question mark means that the content is in quote encoding. quote encoding is the conversion of each byte of the content into a corresponding hexadecimal string. For example, the utf-8 encoding of “Zhang San” is 0xE5BCA0E4B889, so the conversion to quote encoding is =E5=BC=A0=E4=B8=89. The equal sign here is specified by MIME. Some places also use the percentage sign.

Look at the Subject field. utf-8 is followed by ?b?, which means that the content after it has been processed in base64 encoding. The rest of the fields without Chinese are no different from the previous English emails. The following is the Content-Type and Content-Transfer-Encoding. This part is very similar to the HTTP protocol, which specifies the encoding format and transmission format of the email body. The Content-Type is text/plain; charset=utf-8, which means the body is plain text in utf-8 encoding. And Content-Transfer-Encoding is quoted-printable, which means that the body of the message will be converted to quote encoding.

This is the general principle of using email to send non-ASCII content. We certainly don’t need to handle such complex encoding operations ourselves. I found an open source library enmime and the whole process was simplified to

m := enmime.Builder().
  From("张三", "z3@foo.com").
  To("涛叔", "hi@taoshu.in").
  Subject("邮件摘要").
  Text([]byte("邮件正文"))

I have combined go-imap with enmime to enable emailing myself via imap protocol (comment).

Finally we prepare a form to receive the content of the user’s message, with the following fields

<form onsubmit="event.preventDefault(); reply()">
  <textarea name="content" placeholder="留言" required></textarea>
  <input name="email" type="email" placeholder="邮箱（可选）">
  <input name="name" type="text" placeholder="名字（可选）">
  <input type="submit" value="提交">
</form>

The form is submitted using AJAX with the following source code.

function reply() {
  var data = new FormData(event.target);
  data.append("subject", document.title);
  fetch('/+/mail', {
    method: 'POST',
    body: new URLSearchParams(data),
  })
  .then(resp => alert("留言提交成功"))
  .catch(error => alert("接口报错："+error));
}

When the server receives the data submitted by the user, it automatically generates a MIME email and then uploads it to the inbox via IMAP. This way I can receive the notification at the first time. If the user himself left an email address, I can also directly reply to the email.

The above is all about using IMAP to implement message board. The code is also published to GitHub. You can leave me a message now. This approach does not require you to register an account, which is especially friendly to those who pay special attention to privacy protection. And, more importantly, when the author can receive messages by configuring the email client, in learning convenience. But this way does not support real-time display of user comments, which is a pity. However, I think the comments of the blog should be mainly to serve the communication between readers and authors, and the discussion between readers should not be the main conflict. Let’s go online first and see the effect. Another advantage of not showing real-time comment content is to dissuade those who want to send advertising messages. Because they can’t see it, there is no need to send it.

Table of Contents