How and why we built Masked Email with JMAP – an open API standard

Enter JMAP: the developer-friendly, open API standard for modern mail clients. Below, we’ll introduce you to JMAP, explain why we chose it for Masked Email, describe how the integration works, and share how you can get started using JMAP in your own projects.

I’ll be honest, I’d never heard of JMAP (JSON Meta Application Protocol) before we started working on the proof of concept for Masked Email. I was amazed that I’d never heard of this standardized open protocol (RFC8620) that can do so much – JMAP is faster than its predecessors, it’s an open standard, and it’s easy to use.

The more I read about JMAP, the more I realized how antiquated the de-facto APIs that run our digital lives are. The parts of the internet you use every day run on technology from the 80s and 90s – let that sink in for a moment. Remember (or please imagine, young Gen Z) how slow things were in the 80s and 90s? Does the scratching, beeping sound of dial-up make you itch? It was slow, and if your older sister got a call from one of her friends, well… goodbye, internet. Ever noticed how email on your phone takes longer to load than other things on the mobile web? The side-by-side comparison of loading email on iPhones over JMAP versus over IMAP highlights just how outdated email technology has become.

If you’ve ever worked on legacy code, you’ll know how hard it is to add new features, or to just make things better. The languages and libraries you use every day have changed immensely since you started using them. React, for example, was released in 2013, and the API docs today look nothing like they did then. By contrast, IMAP (Internet Message Access Protocol), the tech that helps bring you Gmail (and basically all mail), was basically finalized in RFC3501 in 2003 – 18 years ago. The top-grossing films that year were Finding Nemo, the first Pirates of the Caribbean, the third installment of The Lord of the Rings, and the second Matrix movie. A good year, but technology has progressed immensely since then – the iPhone didn’t even exist yet!

All the things you can do with IMAP and CalDAV (the current standard for calendar sync), you can do more easily with JMAP. From a tech perspective, it feels very familiar. As the name suggests, JMAP is based on JSON and HTTP, some of the first concepts you learn as a developer. With JMAP, you can batch actions together to cut down on the number of requests going back and forth – a big part of why it’s faster. In plain terms: JMAP is made to sync information from where the data is stored (a server) to where you’re using or viewing it (a client), quickly.

To take a closer look at what I’m talking about, let’s review some sample code from our Fastmail friends.

If we boil it down, the two core concepts of JMAP are the Session object and structured data exchange.

The Session

The Session object is the first thing you grab when you authenticate. It tells you everything you need to know about the server, including the maximum supported file size, request, number of calls, which email accounts are connected to that larger account, and even the URL (apiUrl) you’ll need to request from to fetch and sync data via structured data exchange.

Here’s how you authenticate to grab your own Session object in JavaScript.

Let’s say you have a file called jmap-session.js with the code below:

const fetch = require("node-fetch");

/* bail if we don't have our ENV set: */
if (!process.env.JMAP_USERNAME || !process.env.JMAP_PASSWORD) {
 console.log("Please set your JMAP_USERNAME and JMAP_PASSWORD");
 console.log(
   "JMAP_USERNAME=username JMAP_PASSWORD=password node hello-world.js"
 );

 process.exit(1);
}

const hostname = process.env.JMAP_HOSTNAME || "jmap.fastmail.com";
/* your Fastmail email */
const username = process.env.JMAP_USERNAME;
/* your Fastmail "App Password", *not* your real password.
Generate one by going to Settings > Password & Security > App Passwords > Enter your password and click Unlock, and then New App Password and follow the steps listed. More ➡️ https://www.fastmail.help/hc/en-us/articles/360058752854-App-passwords */
const password = process.env.JMAP_PASSWORD;

const auth_url = `https://${hostname}/.well-known/jmap`;
const auth_token = Buffer.from(`${username}:${password}`).toString("base64");

const getSession = async () => {
 const response = await fetch(auth_url, {
   method: "GET",
   headers: {
     "Content-Type": "application/json",
     Authorization: `basic ${auth_token}`,
   },
 });
 return response.json();
};

getSession().then((session) => {
 console.log(session);

 /* TODO(you) cool stuff with your email */
});

In your terminal, run the following using your Fastmail app password in the password input:

JMAP_USERNAME=nobody@fastmail.com JMAP_PASSWORD=4kJc7vuRVwyLKhKF node jmap-session.js

Here’s what the output should look like:

{
  state: 'cyrus-218;p-10',
  uploadUrl: 'https://jmap.fastmail.com/jmap/upload/{accountId}/',
  accounts: {
    ue5494021: {
      isArchiveUser: false,
      isPersonal: true,
      isReadOnly: false,
      name: 'you@fastmail.com',
      accountCapabilities: [Object],
      userId: '17597792'
    }
  },
  username: 'nobody@fastmail.com',
  apiUrl: 'https://jmap.fastmail.com/jmap/api/',
  eventSourceUrl: 'https://jmap.fastmail.com/jmap/event/',
  primaryAccounts: {
    'urn:ietf:params:jmap:submission': 'ub0b940a2',
    'urn:ietf:params:jmap:mail': 'ub0b940a2',
    'urn:ietf:params:jmap:vacationresponse': 'ub0b940a2',
    'urn:ietf:params:jmap:core': 'ub0b940a2'
  },
  capabilities: {
    'urn:ietf:params:jmap:submission': {},
    'urn:ietf:params:jmap:mail': {},
    'urn:ietf:params:jmap:vacationresponse': {},
    'urn:ietf:params:jmap:core': {
      collationAlgorithms: [Array],
      maxObjectsInGet: 4096,
      maxSizeRequest: 10000000,
      maxSizeUpload: 250000000,
      maxConcurrentUpload: 10,
      maxCallsInRequest: 50,
      maxObjectsInSet: 4096,
      maxConcurrentRequests: 10
    }
  },
  downloadUrl: 'https://beta.fastmailusercontent.com/jmap/download/{accountId}/{blobId}/{name}?type={type}'
}

There’s lots of useful information here, but for now let’s focus on apiUrl and primaryAccounts. Here’s where you can grab your account ID. Your account ID is the value at the key urn:ietf:params:jmap:mail. – in this case, ub0b940a2. You’ll need both your apiURL and account ID to do anything fun.

Structured Data exchange

Now let’s talk JMAP API requests. The request and response type is always application/json. The body of every JMAP request, across the board, contains a using and a methodCalls property. The using property is where you put the capabilities you want to have access to with your request. methodCalls is a list of all the data you want to sync, delete, create, fetch, etc. Each method call item is an Invocation data type and it’s always a tuple with 3 items: the name of the method (query, for example), an object with arguments (i.e. the data you want to create/update/delete) based on the method, and the method call id- an arbitrary string that represents the method call. Say you want to edit your most recent email draft:

const mailboxQuery = async (api_url, account_id) => {
 const response = await fetch(api_url, {
   method: "POST",
   headers: {
     "Content-Type": "application/json",
     /* Here let's say you already have an auth token */
     Authorization: `basic ${auth_token}`
   },
   body: JSON.stringify({
     /* The using param is like the scope or capabilities you want access to. In this case, the core JMAP functionalities (you'll always want this one), and your mailbox. */
     using: ["urn:ietf:params:jmap:core", "urn:ietf:params:jmap:mail"],
     /* methodCalls are the items you want to sync with the server. These are processed in the order you give them- easy! */
     methodCalls: [
       /* Each methodCall is an Invocation data type- a wild tuple. An Invocation consists of a string- the name of the request ("Mailbox/query" here), an object with arguments, and another string- the method call id ("a", in the example below). Here we're creating a mailbox query, asking for all of the emails in a user's drafts folder. */
       [
         "Mailbox/query",
         { accountId: account_id, filter: { role: "drafts" } },
         "a"
       ]
     ]
   })
 });
 const data = await response.json();

 return await data["methodResponses"][0][1]["ids"][0];
};

After grabbing the returned ID of your draft folder, you make a third request with the content of your draft email using the same basic format. This time, we’ll have two method calls: one to create the draft, and the other to initialize and send the email:

const draftResponse = async (api_url, account_id, draft_id) => {
 /* The content of your email! */
 const message_body =
   "Hi! \n\n" +
   "This email may not look like much, but I sent it with JMAP, a new protocol \n" +
   "designed to make it easier to manage email, contacts, calendars, and more of \n" +
   "your digital life in general. \n\n" +
   "Pretty cool, right? \n\n" +
   "-- \n" +
   "This email sent from my next-generation email system at Fastmail. \n";

 const draft_object = {
   from: [{ email: username }],
   /* A little to me, from me, for practice 🙂 */
   to: [{ email: username }],
   /* The subject line in your email */
   subject: "Hello, world!",
   /* Yes, we're updating an existing draft */
   keywords: { $draft: true },
   mailboxIds: { [draft_id]: true },
   bodyValues: { body: { value: message_body, charset: "utf-8" } },
   /* We'll stick with plain text emails for now, but if you like HTML you can have a lot of fun with this */
   textBody: [{ partId: "body", type: "text/plain" }],
 };

 const response = await fetch(api_url, {
   method: "POST",
   headers: {
     "Content-Type": "application/json",
     Authorization: `basic ${auth_token}`,
   },
   body: JSON.stringify({
     using: [
       "urn:ietf:params:jmap:core",
       "urn:ietf:params:jmap:mail",
       /* Now we're adding the submission capability to send the email */
       "urn:ietf:params:jmap:submission",
     ],
     methodCalls: [
       /* Now we've got 2 methodCalls */
       [
         /* One to update the draft email */
         "Email/set",
         { accountId: account_id, create: { draft: draft_object } },
         "a",
       ],
       [
         /* And one to create and send the email */
         "EmailSubmission/set",
         {
           accountId: account_id,
           onSuccessDestroyEmail: ["#sendIt"],
           create: { sendIt: { emailId: "#draft" } },
         },
         "b",
       ],
     ],
   }),
 });
 const data = await response.json();

 console.log(JSON.stringify(data));
};

And that’s it! To do the same thing in IMAP, you’d actually need to use both IMAP and SMTP (Simple Message Transfer Protocol); IMAP to receive email and SMTP to send it. Or, you can learn JMAP once and use it everywhere, for anything.

JMAP & Masked Email

JMAP has been designed to be used as a protocol on top of any kind of server/client communication, it’s not just specific to email or Fastmail – for example, an API to mask your real email! We at 1Password got early access to the Masked Email API, and soon Fastmail will be opening it up to everyone so you can build your own Masked Email integration.

“We wanted to build our feature on open standards because anyone can use them and the potential audience for reuse is huge. OAuth was an obvious choice, but using JMAP was an exciting choice. Using it is more proof that the protocol has a lot of uses beyond just email, and we’re excited to keep using it for new features.” - Ricardo Signes, Fastmail CTO

The Masked Email API follows the same format you see above. When the Fastmail team was done building the API, all we had to do was update the method calls to look like this snippet Fastmail CEO Bron Gondwana posted on Hacker News:

Naturally, our team did a ton of other development to realize Masked Email in 1Password, but the JMAP requests were the easiest part because they all followed the same core protocol that we were already familiar with.

Because of its flexibility, speed, open standards, and ease of use, JMAP was the perfect tool for Fastmail to develop the Masked Email API, and we had a blast learning it. We believe technology should improve based on the needs of its users. If you build something with your Fastmail account and JMAP, we want to see it!

Ready to get started?

Here are all the things we watched, read, and played around with to learn JMAP:

• The official JMAP docs: https://jmap.io/index.html
• Fastmail’s JMAP-Samples GitHub repo: https://github.com/fastmail/JMAP-Samples
• Rik’s talk at last year’s Technical.ly Developers Conference: https://www.youtube.com/watch?v=5As3E8mXiMA
• Rik’s JMAP Crash Course on Topicbox: https://jmap.topicbox.com/groups/fastmail-dev-beta/T83594f41ca76f56c/jmap-crash-course
• Fastmail’s Digital Citizen podcast episode: Why Open Internet Standards Are So Important To Your Future with Bron Gondwana: https://www.fastmail.com/digitalcitizen/why-open-internet-standards-are-so-important-to-your-future-with-bron-gondwana/transcript/

Madeline Hanley

Lead Engineer, Service Integrations