Cypht Webmail (aggregator): Add JMAP support (as an alternative to IMAP + SMTP) and permit unified inbox (and even move or copy emails from one account to another)

[marclaporte]

"JMAP is intended to be a new standard for email clients to connect to mail stores. It therefore intends to primarily replace IMAP + SMTP submission. It is also designed to be more generic such that it can be extended with contacts, calendars in the future (replacing CardDAV/CalDAV). It does not replace MTA-to-MTA SMTP transmission."

http://jmap.io/

[jasonmunro]

Thanks for the feedback. I have heard of jmap before, and briefly reviewed it a while back, but I'm not super familiar with the specification. Could be a cool addition to Cypht as either a stand alone module set or as part of the imap modules.

[marclaporte]

Super!

Someone else asked me "Who uses or supports it?" My answer copied below:

Cyrus IMAP and Apache James are the notable ones here: http://jmap.io/software.html

OpenPaaS is an open source Entreprise Social Platform which is led by LINAGORA (which has more than 150 employees)
https://linagora.com/
http://www.open-paas.org

OpenPaaS is using Apache James. And you can see JMAP used in the code:
https://github.com/linagora/openpaas-esn/search?utf8=%E2%9C%93&q=jmap&type=

In this video: https://www.youtube.com/watch?v=2QZIi3EO6rQ , Benoit Tellier explains why they decided to invest in James and JMAP (among other things). LINAGORA has 5 full time employees which are devoted to the project.

Why LINAGORA picked JMAP:
https://medium.com/linagora-engineering/you-cant-do-imap-in-the-browser-can-you-208838b0ba17

FastMail started this initiative:
"JMAP is FastMail's protocol with the warts removed. We leverage existing standards like HTTP, JSON, and native push channels on platforms which have them - making it easy for developers to work with."
https://blog.fastmail.com/2016/12/22/cyrus-development-and-release-plans/
https://blog.fastmail.com/2016/12/12/why-we-contribute/
https://blog.fastmail.com/2016/12/09/jmap-test-suite/
https://blog.fastmail.com/2015/12/23/the-jmap-momentum-builds/
https://blog.fastmail.com/2015/08/11/an-open-source-jmap-proxy-javascript-library-and-webmail-demo/
https://blog.fastmail.com/2014/12/23/jmap-a-better-way-to-email/

Roundcube Next will support JMAP as well.
https://github.com/search?q=org%3Aroundcube-next+jmap&type=Code

Thanks!

[jasonmunro]

Thanks for that info. I took another peek at the JMAP spec after you submitted this. Looks like a pretty straight forward rest-style api. Cypht was designed to aggregate data from different sources, so a lot of the work of uncoupling from a particular api or protocol is baked in already. Having worked with the IMAP protocol for 15+ years, I understand more than most why something like JMAP is needed :) I think it differs enough from IMAP that it would make sense to create a stand alone module set (also the IMAP module set is already a tremendous amount of code).

I can't promise I will get to this real soon, but I am intrigued. I suspect initial support for this would entail a similar amount of work that the Github module set took - I also suspect we could abstract various bits of the existing IMAP code where the two protocols might overlap.

[marclaporte]

Dear Jason,

Now is a good time to start the implementation:
https://fastmail.blog/2018/12/27/jmap-is-on-the-home-straight/

Thanks!!!

[dumblob]

@marclaporte how I see it, the most important information is, that during the upcoming weeks/months, JMAP will finally become an IETF standard. Which is a huge step from my point of view, but seems to be also a huge step forward form the JMAP developers point of view (the IETF processes increased the JMAP quality significantly).

[jasonmunro]

@dumblob @marclaporte I agree and that was an interesting read, thanks for the link. It's a new year and I'm feeling ambitious so I'm going to bump this up and see what I can get done. The other new feature in the works is cardDav, which has read only support now, and I want to expand that to read/write then move on to calDav support. No reason we can't push both that and JMAP forward at the same time :)

One prerequisite for this is I want to create a "skeleton" module set for new data sources. Sort of a template of all (or at least most) of the entry points required to create a new data source inside Cypht. Then I can use that as the foundation for supporting JMAP, as well as making it easier to build support for any other protocol we want to implement.

[ulfgebhardt]

A slim PHP Library is missing at current time.

There is one for Zend:
https://github.com/WikiSuite/zend-jmap

There is many for JS (which is not compatible with current architecture of cypht?!)
https://github.com/search?l=JavaScript&q=jmap&type=Repositories

Protocol Meta for Client:
https://github.com/jmapio/jmap/blob/master/client-guide/jmap-client-guide.mdown

I think it is possible to extract the Implementation out of the Zend Framework from here:
https://github.com/WikiSuite/zend-jmap/blob/master/src/Jmap.php
(and parent folder respectively)

But the License is quite limiting:
https://github.com/WikiSuite/zend-jmap/blob/master/LICENSE.md

[dumblob]

For the record, there is also a JMAP<->IMAP proxy which could be useful for testing of some corner cases.

[jasonmunro]

@ulfgebhardt and @dumblob thanks for the resources! I suspect we will be writing our own JMAP library in PHP for this. Once we have the "skeleton" module set in place for data sources, most of the boilerplate required to set this up in Cypht is in place. Since JMAP is very RESTful, we can leverage our existing API wrapper code that makes doing RESTy things easy. The final bit is the actual work of deciphering the spec and building out the parsing and logical flows which I suspect won't be nearly as hard as IMAP. We also have a nice set of functions in the core module set for common operations so we can just re-use things we use now (like decoding mime encoded header values for example).

[jasonmunro]

As I dig into this I think I might put JMAP support into the IMAP and SMTP module sets. It will likely be a LOT easier with the only downside being you need those 2 sets enabled to use JMAP.

After much head scratching and trial and error, I have a git checkout of Cyrus IMAP running locally with JMAP support - which should be the most up to date WRT JMAP specs as we can get :)

[jasonmunro]

Update: spent some time on this over the weekend. Some random thoughts:

• It's a pretty cool API. Docs have been spot on so far and the design is consistent
• It's not as "folder" based as IMAP. Messages are still grouped by folders and folders have hierarchy of course, but message uuids are globally unique so there is no need to select a folder before accessing a message
• Performance is great, though it IS Cyrus IMAP running locally so I would expect that :)
• I started hacking around by swapping out the public interface to our IMAP lib with a matching JMAP version. This has been ... interesting. It's a hell of a lot less duplicate code than creating a stand alone module set, and maybe it could be sort of a "drop in" replacement. But It's quickly becoming obvious the protocol differences are going to require some better abstraction at the module layer that interacts with the library for that to work :)
• Pagination support for message lists!
• Our API wrapper for http requests using curl is working out nicely for this. No changes required so far.
• Managed to cobble together folder navigation. Barely tested but it's functional.

After getting folder navigation going I tried to add old school mailbox browsing, but can't seem to get quite what I need yet. Surely I'm doing something wrong but I filed an issue anyway because whatever is not working I just don't see it yet: cyrusimap/cyrus-imapd#2627

[jasonmunro]

As expected the issue I was having with Cyrus JMAP was not a bug, but a configuration issue on my end. With there help we are back in business :)

[jasonmunro]

Update: Now able to browse mailboxes and have a basic message view working. Here is the roadmap to MVP:

• Finish the hm-jmap.php lib to include all the methods the IMAP lib uses
• Add a new section on the servers page to add a JMAP server
• incorporate JMAP into the SMTP module set for outbound mail
• include JMAP as a possible IMAP or SMTP server in profiles

Once we get that far, I want to circle back around to the library code and see what can be done to improve the performance. Right now we are using JMAP in a very IMAP like way, but we can streamline that in a number of places to take advantage of JAMP features like multiple "commands" in a single API call.

[jasonmunro]

Update: Ran into another problem with Cyrus JMAP that maybe is not me doing something wrong :) filed an issue (cyrusimap/cyrus-imapd#2631).

In the meantime, Cypht JMAP support now works with combined views (unread, flagged, everything). I also reworked the folder logic in the library and was able to remove some module level work-arounds that were icky. Then I started on search integration but was thwarted by the above issue.

JMAP is complicated (E-mail is complicated!), but it truly is IMAP simplified and modernized. There is lots more to do but we are making great progress a lot faster than I expected. And I'm sort of obsessed with making it feature complete now that I started working on it :)

[jasonmunro]

Update: Wrong again. My latest issue was me doing something wrong (specifically not configuring the Cyrus search backend properly). Thanks to the helpful feedback from Cyrus devs it is resolved and search works now with JMAP, as does some message operations (flag/read/unread/etc). For core IMAP stuff needing to be converted, I have move, copy, delete, and folder management up next, following by downloading and then I think we are pretty close to IMAP feature complete!

[jasonmunro]

Compatibility layer for JMAP is feature complete! Needs more testing and love but it's working pretty well so far. Here is the code:

https://github.com/jasonmunro/cypht/blob/master/modules/imap/hm-jmap.php

Comparing this to the IMAP lib:
IMAP uses 3036 lines of code excluding comments
JMAP uses 861 lines of code excluding comments

WOW! Up next is wiring the UI to add/delete JMAP servers from the Settings -> Servers page.

[jasonmunro]

UI update to add/delete JAMP servers is pushed to the master branch :) Next I'm going to scope out using JMAP for SMTP. This should be way easier as we only communicate with SMTP in 2 places of the code if I recall.

[dumblob]

Is anyone of you in contact with JMAP developers or just people around JMAP?

It seems Cypht will be one of the very few first clients supporting JMAP - that could be used to raise awareness of Cypht and its security and mobile advantages. We could start by letting Cypht being linked from JMAP websites. Then maybe write a blog post on JMAP sites, etc.

[jasonmunro]

@dumblob I have been in contact with the JMAP folks, and we are now listed here: https://jmap.io/software.html. Also I wrote a blog post on JMAP that made it to the front page of hacker news last night and it's driving a TON of traffic :)

https://unencumberedbyfacts.com/2019/01/24/jmap-its-like-imap-but-not-really/

[marclaporte]

Bron Gondwana presented JMAP at FOSDEM 2019.

Slides with Speakers' Notes:
https://fosdem.org/2019/schedule/event/email_standards/attachments/paper/3116/export/events/attachments/email_standards/paper/3116/jmap_slides_fosdem_2019_notes.pdf

General page: https://fosdem.org/2019/schedule/event/email_standards/

Well done Bron!

And well done Jason for such a quick implementation!

[OJFord]

For the benefit of others linked to this issue from the JMAP site, my understanding is that this issue remains open only because the JMAP feature isn't yet in a (non-RC) release?

I.e. it is in master, and the (at time of writing) latest 'release', v1.2.0-rc1.

I look forward to trying it, thanks!

[marclaporte]

@OJFord : yes, that is correct. And as this is a new feature, on a new protocol, expect it to be bumpy, and please file new issues for bugs and missing features.

I encourage you to get Cypht master and start testing it out. I feel master is stable enough to be released.

In Tiki, we get Cypht master from Packagist and it works quite well:
#311

If you have trouble installing Cypht master, you can install a daily build of Tiki
20.x is stable with Cypht (I use this daily with 3 mail accounts)
21.x is not released yet and has CalDAV / CardDAV as well (also expect this to be bumpy, and UI is still uncoordinated)
https://dev.tiki.org/Daily-Build

[landryb]

Fixed the above by adding JSON_UNESCAPED_SLASHES flag to the json_encode call in format_request... now the requests sent to the jmap server all return 200 codes, but cypht returns a 500 to the user. making progress slowly :)

[marclaporte]

@Yannick243 will resume work here, with some guidance from @josaphatim

[marclaporte]

@Shadow243 now joins @Yannick243

[marclaporte]

#909

[marclaporte]

Progress:

• Fixed error when adding JMAP server #636
• Permit to turn off JMAP #658
• improve JMAP compatibilities with stalwart #909

[marclaporte]

Spin off: #931

[marclaporte]

@Shadow243 @Yannick243 What is the status here? It's time to release Cypht 2.0 :-)

[Shadow243]

@Shadow243 @Yannick243 What is the status here? It's time to release Cypht 2.0 :-)

Working on this task, we encountered some issues that we had to fix first: #936

We are currently working on the second PR, which will only cover JMAP.

[landryb]

i'm feeling a bit dumb, but resuming testing on this against stalwart 0.6.0, in the current cypht master i haven't found a way to setup a new jmap account, there's only IMAP and SMTP listed once you've specified your email/passwd. Is there some hidden setting somewhere ? And i dont feel like #939 is fixing it, since it seems to be for an already configured jmap account. the jmap module is correctly listed in CYPHT_MODULES in .env

[Shadow243]

i'm feeling a bit dumb, but resuming testing on this against stalwart 0.6.0, in the current cypht master i haven't found a way to setup a new jmap account, there's only IMAP and SMTP listed once you've specified your email/passwd. Is there some hidden setting somewhere ? And i dont feel like #939 is fixing it, since it seems to be for an already configured jmap account. the jmap module is correctly listed in CYPHT_MODULES in .env

[landryb]

i'm feeling a bit dumb, but resuming testing on this against stalwart 0.6.0, in the current cypht master i haven't found a way to setup a new jmap account, there's only IMAP and SMTP listed once you've specified your email/passwd. Is there some hidden setting somewhere ? And i dont feel like #939 is fixing it, since it seems to be for an already configured jmap account. the jmap module is correctly listed in CYPHT_MODULES in .env

ah, thanks. i dont have that checkbox, but now i see where it comes from in the code, so i'll try to figure out which bit is missing.

edit bah, nevermind, found it.

[marclaporte]

@landryb I am super happy you are helping with JMAP testing.

General dev activity level on Cypht has been good recently: https://openhub.net/p/cypht

We have many new features and enhancements coming to Cypht 2.0, and JMAP support is the last one I am hoping we can add: #879

All the active devs are new to JMAP so it's much more challenging than other enhancements.

Thanks!

[landryb]

my testing has been limited so far, but im mostly struggling with stalwart, not that much with cypht after all :) Been also testing jmap with meli...

[marclaporte]

@landryb Are you saying that JMAP-Cypht is mostly working and we can release Cypht 2.0?

[landryb]

i've been able to connect to a JMAP testing account without specific subfolders (eg only default folders), and go to folders that contained mails (#939 was needed for that). The number next to the folder name was always zero, regardless of the folders content.

i havent tested folders/subfolders nor large volumes of e-mail, nor sending an e-mail.

[marclaporte]

Cypht 2.0 will ship with JMAP support, with at least one thing missing (will be done after 2.0): #931

[marclaporte]

Cypht 2.0 has been released!
https://github.com/cypht-org/cypht/releases/tag/v2.0.0

[marclaporte]

As of now, we only have one open issue related to JMAP within Cypht: #931

In theory, everything else works. Since this is so new, I do expect we'll encounter issues and we will work in priority on all issues related to JMAP. So please test!

Our reference dev/test server is Stalwart JMAP, and we'd love to collaborate with the Cyrus and Apache James communities, and anyone else working on JMAP!

[marclaporte]

Here is a somewhat related thread:
https://forum.virtualmin.com/t/a-mail-server-that-supports-jmap-json-mail-access-protocol-to-replace-imap-smtp-carddav-and-caldav/126214/3

[marclaporte]

As of now, there are 3 webmail clients listed here: https://jmap.io/software.html. If others exist, please update https://github.com/jmapio/jmap/blob/master/software/software.mdown

Of the 3:

• one is a demo
• one uses JMAP for an API: Added Group-Office jmapio/jmap#293
• the third one is Cypht

So I conclude Cypht is the first/only Open Source standalone webmail client supporting JMAP. While it’s nice to be the trailblazer, I hope many others will join the club 😊. JMAP is an open standard. We can all work together to make it better. Specifically, I hope Roundcube will join the JMAP club soon (I made the original feature request in 2017): roundcube/roundcubemail#5918

Here are other requests I am tracking:

• SnappyMail: JMAP support the-djmaze/snappymail#1550
• Nextcloud Mail: JMAP support nextcloud/mail#2931
• Rainloop: Add JMAP support (as an alternative to IMAP + SMTP) RainLoop/rainloop-webmail#1378 (I made the original feature request in 2017)
• Claws Mail: https://www.claws-mail.org/bugzilla/show_bug.cgi?id=4057 (I made the original feature request in 2018)
• Thunderbird: https://bugzilla.mozilla.org/show_bug.cgi?id=1322991
• K-9 Mail (Thunderbird for Android): [EPIC] JMAP protocol support thunderbird/thunderbird-android#3272
• Evolution: https://gitlab.gnome.org/GNOME/evolution/-/issues/364
• DavMail: Add support for JMAP (JSON Meta Application Protocol), as an alternative to IMAP + SMTP mguessan/davmail#365 (I made the original feature request in 2024)

Beyond JMAP, we are now working on EWS (Exchange Web Services) support: #1278

On a related note, here is a page to mainly compare all active and mature Open Source (thus self-hostable) webmail solutions to manage email from any server with standard email protocols (IMAP, SMTP, JMAP) and EWS (Exchange Web Services):
https://wikisuite.org/Webmail-and-groupware-comparison

Cypht permits to have all emails in a unified inbox, and even copy or move emails between accounts.