ProtonMail Bridge IMAP integration

[stellarsquall]

Using the newly released ProtonMail Bridge I am unable to add an IMAP account, with the error:

"Sorry, this folder does not exist. (IMAP)"

Log contents:

----------IMAP----------

* OK [CAPABILITY IMAP4rev1 STARTTLS AUTH=PLAIN AUTH=LOGIN ID APPENDLIMIT] IMAP4rev1 Service Ready
1 STARTTLS
1 OK Begin TLS negotiation now
2 LOGIN "[email protected]" *********
* CAPABILITY IMAP4rev1 AUTH=PLAIN AUTH=LOGIN ID XSTOP APPENDLIMIT
�2 OK [CAPABILITY IMAP4rev1 IDLE MOVE SPECIAL-USE ID XSTOP QUOTA APPENDLIMIT=26214400] LOGIN completed
3 LIST "" ""
* LIST (\Noselect) "/" "/"
3 OK LIST completed
4 ID NIL
* ID (support-url https://protonmail.com/support name ProtonMail vendor "Proton Technologies AG")
4 OK ID completed
ID (support-url https://protonmail.com/support name ProtonMail vendor "Proton Technologies AG")
����5 LIST "" "*"
�H�Y|�� 

The Bridge is required to provide a decryption mechanism for ProtonMail. I've followed the instructions that might apply to a Thunderbird account, found here:

https://protonmail.com/bridge/thunderbird-mac#3e

Are there any related issues?

Question was originally asked in closed Issue #367, with another user experiencing the same issue.

What operating system are you using?

OSX 10.11.6

What version of Mailspring are you using?

Version 1.0.10 (1.0.10)

--

Bug?

Do you have any third-party plugins installed? If so, which ones?

None

Is the issue related to a specific email provider (Gmail, Exchange, etc.)?

ProtonMail Bridge

Feature Request?

Does this feature exist in another mail client or tool you use?

Might be a feature request, considering Mailspring seems to think that there aren't any selectable folders in the account. The bridge appears to be connecting as expected, and the Thunderbird works as expected.

[bengotow]

Hey! Thanks for reporting this - when you view the account in thunderbird, does it show any sort of folders or "namespaces" for the account? Basically Mailspring is requesting a list of folders and getting nothing back:

3 LIST "" ""
* LIST (\Noselect) "/" "/"
3 OK LIST completed

I suspect we need to change what we're asking for somehow, but I'm not sure.

[stellarsquall]

Thanks for the response @bengotow!

I'm now starting to understand the log. In the sidebar for thunderbird I see the following account heirachy:

Inside of thunderbird's account settings for local folders, I can browse the location of the local folders on disk and see the following:

[durch]

+1 for this it would be great to have it, I currently have ProtonMail bridge connected to Apple Mail, and can help troubleshooting/testing...

Attaching log for completness

----------IMAP----------
* OK [CAPABILITY IMAP4rev1 STARTTLS AUTH=PLAIN AUTH=LOGIN ID APPENDLIMIT] IMAP4rev1 Service Ready
1 LOGIN "*********" *********
1 OK [CAPABILITY IMAP4rev1 IDLE MOVE SPECIAL-USE ID QUOTA APPENDLIMIT=26214400] LOGIN completed
2 LIST "" ""
* LIST (\Noselect) "/" "/"
2 OK LIST completed
3 ID NIL
* ID (vendor "Proton Technologies AG" support-url https://protonmail.com/support name ProtonMail)
3 OK ID completed
ID (vendor "Proton Technologies AG" support-url https://protonmail.com/support name ProtonMail)

3 OK ID compl4 LIST "" "*"
�

[bengotow]

Hey folks - thanks for the extra info. I've emailed protonmail to see if they can help investigate this (since Mailspring uses Mailcore2, which is pretty popular, this might impact other clients as well.) I'll see if I can reproduce the issue and keep you posted.

[bengotow]

Just a quick update about this. I signed up for a paid protonMail account and setup the IMAP Bridge. It looks like Mailspring / Mailcore2 is unable to parse ProtonMail's response to the ID command, which causes failures downstream when it tries to make the next command (LIST).

Here's ProtonMail's response alongside a readable response from Gmail:

Works:
* ID ("name" "GImap" "vendor" "Google, Inc." "support-url" "https://support.google.com/mail" "version" "gmail_imap_171205.10_p0" "remote-host" "2601:640:4:b0f7:c175:9442:6f47:7ad3")\r\n

Fails:
* ID (support-url https://protonmail.com/support name ProtonMail vendor "Proton Technologies AG")\r\n

I think the problem is that ProtonMail's response isn't enclosing the values in quotes. Looking over the spec (https://tools.ietf.org/html/rfc2971#ref-IMAP4rev1) it's unclear whether the quotes are supposed to be required (though all the examples show quotes). I think it'd be a good idea for ProtonMail Bridge to change this behavior, but in the meantime I'll see if I can fix the parsing issue or find a workaround.

Update: It looks like the parsing issues extend beyond the ID command - Mailcore2 can't parse the FETCH responses either. I think the issue is still missing quotes, but I'm hesitant to make changes to the parsing logic because it's been extensively tested against pretty much all IMAP providers. It's probably a better idea to wait and see if ProtonMail changes Bridge to behave more like a typical IMAP server.

[stellarsquall]

Thanks for your work on this @bengotow.

It appears that ProtonMail Bridge 1.0.1 now allows me to add the account. However I'm not able to import any messages, with the "status" for the account stating "Scanning messages". The scanning attempt eventually times out with "Encountered an error while syncing" the account.

Attached is the log for reference.

mailsync-6cd75d34.log

[ploum]

Hi @bengotow,

If you don't want to pay, I can offer you a mail account under one of my domain which is hosted by ProtonMail.

As a contribution,  I also offer to reimburse what you had to pay only to test this bug.

[bengotow]

Hey! Thanks for the reply—I've actually signed up for a paid ProtonMail subscription and reproduced the problem locally, but after doing the investigation above I've decided to wait to see if the ProtonMail folks fix this. I think they should, as this probably breaks other clients as well. The parsing code we'd need to change in mailcore2 is /very/ well tested with other providers but has no specs, so I'm fairly afraid of breaking other scenarios I don't know about in the process of fixing this.

[ploum]

You will be interested by https://www.reddit.com/r/ProtonMail/comments/7nn1vh/what_are_the_mail_clients_compatibles_with_the/

Anyway, my offer to have a ProtonMail account on one of my domain stand if you need it later and you cancelled your subscription.

[kkinder]

Perhaps unrelated, when I try this, I'm seeing that Mailspring is having trouble creating the folders it wants (too?):

12946 [2018-01-25 09:39:08.200] [metadataExpiration] [info] Scanning for expired metadata
12946 [2018-01-25 09:39:08.201] [metadataExpiration] [info] -- Will wake for next expiration in 7200sec
12946 [2018-01-25 09:40:53.306] [background] [info] Syncing folder list...
14242 [2018-01-25 09:40:53.538] [main] [info] Identity created at 1515191720 - using ID Schema 0
14242 [2018-01-25 09:40:53.539] [main] [info] ------------- Starting Sync (REDACTED) ---------------
14242 [2018-01-25 09:40:53.542] [background] [info] Marking all folders as `busy`
14242 [2018-01-25 09:40:53.542] [metadata] [info] Metadata delta stream starting...
14242 [2018-01-25 09:40:53.543] [background] [info] Syncing folder list...
14242 [2018-01-25 09:40:53.555] [background] [error] Could not create required Mailspring folder: [Mailspring]/Snoozed. ErrorSubscribe
14242 [2018-01-25 09:40:53.556] [background] [info] Syncing folder list...
14242 [2018-01-25 09:40:53.559] [background] [error] Could not create required Mailspring folder: [Mailspring]/Snoozed. ErrorSubscribe
14242 [2018-01-25 09:40:53.571] [background] [info] syncFolderUIDRange - fetching INBOX, UIDs: 771 - 1521, Heavy: true
14242 [2018-01-25 09:40:53.572] [foreground] [info] Idling on folder INBOX
14242 [2018-01-25 09:40:53.609] [background] [info] Sleeping after exception: {"debuginfo":"syncFolderUIDRange - fetchMessagesByUID","key":"ErrorParse","retryable":true,"what":"std::exception"}

LMK if I can be of any help in futue testing. I'm not sure if the error about creating [Mailspring]/Snoozed is helpful.

[asim-vax]

Not sure if this belongs here or in another ticket - however it's the same as @kkinder
Linux snap mailspring 1.1.4
using the linux beta bridge it can successfully connect to thunderbird, and connects fine in mailspring, pulls all correct folders/labels - however it gets stuck on creating required mailspring folders

25339 [2018-02-20 14:33:27.180] [background] [error] Could not create Mailspring container folder: Mailspring. ErrorSubscribe
25339 [2018-02-20 14:33:27.180] [background] [error] Could not create required Mailspring folder: Mailspring/Snoozed. ErrorSubscribe
25339 [2018-02-20 14:33:27.185] [background] [info] Syncing folder list...
25339 [2018-02-20 14:33:27.186] [background] [error] Could not create Mailspring container folder: Mailspring. ErrorSubscribe
25339 [2018-02-20 14:33:27.186] [background] [error] Could not create required Mailspring folder: Mailspring/Snoozed. ErrorSubscribe
25339 [2018-02-20 14:33:27.191] [background] [info] syncFolderUIDRange for INBOX, UIDs: 1 - 282, Heavy: true
25339 [2018-02-20 14:33:27.199] [background] [info] Sleeping after exception: {"debuginfo":"syncFolderUIDRange - fetchMessagesByUID","key":"ErrorParse","retryable":true,"what":"std::exception"}
25339 [2018-02-20 14:33:27.202] [foreground] [info] Idling on folder INBOX

[uipoet]

@bengotow The current problem is that mailsync attempts to create a top-level Mailspring folder, which Protonmail forbids. The highest level with write access is Folders. So for Mailspring to support Protonmail, you must change the container folder to Folders/MailSpring. Unfortunately, since mailsync is closed-source, I can't learn/provide more context or even submit a pull request to address this issue.

[Tehnix]

Failing to create the folders is also the problem with Airmail support, I believe. If it could either be turned off, or allow the user to manually set it up the folders, that should probably work.

[mitchazj]

I also have this issue, would love to help in any way if I can!

[mitchazj]

Any update here? I would really love to see this integration happen if possible 😇

[ghikio]

Same here! Would love to help on this if needed

[armaccloud]

Here too, I'm using Protonmail Plus + Mailpring. Same errors in the logs.
I have sent a bug report to protonmail for this. I suggest others reading this do it as well.

In the meantime, how much effort would it be to fix it on the mailspring side by changing to /Folders/Mailspring/Snooze in case protonmail is used as a workaround?

[ghost]

I got excited about another mail client option for ProtonMail Bridge until I hit this issue (also stopped me from using Airmail). Please add an option to change the folder used for MailSpring metadata sync or an option to turn it off completely. I will signup for the Pro plan if this issue is fixed...

[joekendal]

Same issue. Looks like this is a simple fix

[rgruesbeck]

It will be a glorious day when this is fixed

[armaccloud]

The bug update from the side of ProtonMail I received doesn't bring much clarity on specific timelines and whether mailspring will be supported unfortunately. Hopefully something will be done on the side of Mailspring.

Hi Arno,

Thanks for the message.

We're aware of this, unfortunately at this time other email clients (other than Outlook, Apple Mail and Outlook) are currently not officially supported, our team is trying to implement support for new email clients.

Let us know if you need anything else.

Best Regards,
The ProtonMail Team

[ams-123]

Has anyone tried with the latest Mailspring?

[asim-vax]

Yep, still the same issue:

mailspring            1.2.2                   216   stable    foundry376

4801 [2018-06-01 11:21:19.146] [background] [error] Could not create Mailspring container folder: Mailspring. ErrorSubscribe
4801 [2018-06-01 11:21:19.147] [background] [error] Could not create required Mailspring folder: Mailspring/Snoozed. ErrorSubscribe

[ewauq]

Checked today, this is not fixed. Please make it happen 🙏

[armaccloud]

I have checked with the new protonmail bridge on Linux that just came out (v1.0.5); it's still not working unfortunately.

[OmarAssadi]

Just ran into this today on the latest version of ProtonMail Bridge for Windows (1.0.5). Would be great to see the folder issue resolved at some point.

[jd327]

Same. Such a shame doesn't work with ProtonMail Bridge...

[exander77]

This is the MailSpring issue, but I could possibly fix their issue in Bridge, but it would need to be accepted by ProtonMail or you would need to use a custom build of Bridge. I modified Bridge to work with K-9 and be run directly on Android phone using Termux for example: https://github.com/exander77/proton-bridge-android

I do not like MailSpring, but if this would be an acceptable solution and you are willing to pay for it...

[ploum]

My contribution to the bounty would gladly go to the person fixing the issue in PM bridge instead of Mailspring.

[exander77]

On the latest versions of the ProtonMail Bridge I've been able to sync, and send mail successfully through Mailspring. Well that is until Mailspring attempts to create its folders in the root directory and throws an error. @bengotow is the Mailspring folder creation something that is a part of Mailspring itself or Mailsync? With the addition of an automatic fall back system to create the folders in a subdirectory or by giving the user the option of folder location in the Mailspring config we might be able to close this issue relatively soon. I'm happy to help out any way I can if the code to help solve this problem is publicly accessible.

That part is deep in Mailsync which is closed source...

[exander77]

@ploum What OS do you use? Are you willing to beta test builds? Ideally, if you could refer back with anything out of the ordinary and will able to describe issues in a reproducible manner. :)

[exander77]

Is anybody willing to test the build?

[YoannMa]

@exander77 I would be willing to do beta testing if needed, I have both a windows and a linux (ubuntu flavor) setup

[exander77]

@YoannMa https://github.com/exander77/proton-bridge/releases/tag/mailspring-0

First try. I need anything which would seem out of the ordinary. I am not a Mailspring user. I tried Mailspring, it shows all folders, the unnecessary Folders prefix disappeared and no errors are present in the log now. Snoozed folder seems to be automatically created. Its location on the server is Folders/Snoozed.

[exander77]

Any other testers?

[YoannMa]

@exander77 I tried out without any issue, snoozing was working as well.

[lsnow99]

@exander77 I am testing on Linux with bleeding edge Mailspring. So far so good, and snoozing seems to work. I am running into an issue, however, where sometimes when returning to the Mailspring window all my emails are gone from the inbox (and it says "Hooray! You're done." when I am selected on the Inbox folder). I can see my emails in the All Mail section but not in the Inbox folder. Could it have something to do with my Inbox folder on protonmail.com being named INBOX instead of Inbox?

I can get the inbox folder to fix itself by doing File -> Sync New Mail Now. Alternatively, waiting a while seems to just resolve the issue (temporarily of course, until it reappears).

Anyone else running into this?

[exander77]

@exander77 Does this happen wich other mail provides?

[exander77]

@YoannMa Please report after more trying. Do you have the same problem as @lsnow99. I am not sure if it is ProtonMail related, his local issue or a general problem with MailSpring.

I think it would be pretty complicated to use custom build version. So I will try to push workaround into ProtonBridge official repo. Probably in a form of command-line option or type --mailspring-workaround or something like that.

[lsnow99]

@exander77 This was happening a few months ago as well with Mailspring on a separate computer I was using. I started following this GitHub issue in hopes the fix to protonmail bridge would fix it. It only happens with my protonmail account. (I have a few gmail and one outlook.com accounts that don't have this issue in Mailspring). I don't see anything relevant in the logs. I might try using a different protonmail account to see if it persists there as well.

[exander77]

@lsnow99 Try that and I will try to reproduce myself.

[lsnow99]

@exander77 Just upgraded a different account for the month for testing and it is having the same issue. Let me know if the issue appears on your account.

[exander77]

@lsnow99 It seems to happen for me as well, from time to time, folder contents disappear, but I can press F5 and they reload. But there is literally nothing related to this in low. No warning, no error. And info messages do not seem to relate.

[lsnow99]

@exander77 Yup that's where I'm at. I'm not sure if fixing that falls under the scope of this ticket or a different one. It does seem to be a protonmail specific incompatibility, however. I also don't know if it can really be addressed without digging into Mailsync which unfortunately seems to be closed source :(

[exander77]

@lsnow99 I try to debug it a little, but I have only communication that goes on with Bridge and logs from Mailspring. I hat Foundry376 that that didn't make Mailsync OpenSource.

Also, their reasoning is total BS:
https://github.com/Foundry376/Mailspring/blob/master/ROADMAP.md#why-is-mailsync-closed-source

All issues could have been easily fixed if it were OpenSource.

I worry it is using either some proprietary code which they do not have rights to use, or it is doing some BS they don't want to be public.

[lsnow99]

Yup. It's looking like I won't continue to use Mailspring unless they open-source Mailsync. Right now mailsync is eating up my CPU in the background- I don't see why it needs to do all this processing if it just needs to sync mail. And considering it doesn't seem compatible with Protonmail, It's looking like I will need to find a new application. Thanks for your help fixing this issue @exander77 .

[exander77]

@lsnow99 There a fork of this project Mailspring Libre, which want to remove anything closed including mailsync, but it will take some time.

[t3pfaffe]

And now this thread has gone full circle...

If any of the Foundry376 devs are wandering down this far it's probably best to close this issue. We're back to the fundamental issue of mailsync being closed source. There's nothing the open source community can do without the source and the devs haven't touched this issue since 2018, so I think it's fair to say that's it for now.

[exander77]

@t3pfaffe I have said it in the beginning that this is Mailsping project issue, not the ProtonBridge issue. If devs were to share sources, I can fix it, I a am not at all worried that we would not figure out how to build it.

I also toyed with the idea to fix it in Mailsync binary. I could rewire Mailspring/Snoozed into Folders/Snoozed. It sounds a bit extreme, but as the mailsync is not updated at all. The problem is that the binary has a lot of occurrences of Mailspring, so I would need to carefully check if that occurrence is relevant. But Mailspring is shorted string than Folders and this should be doable without significant matching.

[forthommel]

Yup. It's looking like I won't continue to use Mailspring unless they open-source Mailsync. Right now mailsync is eating up my CPU in the background- I don't see why it needs to do all this processing if it just needs to sync mail. And considering it doesn't seem compatible with Protonmail, It's looking like I will need to find a new application. Thanks for your help fixing this issue @exander77 .

Sad but absolutely true. Same action for me, will wait for Mailspring Libre to be in an usable state and give up any hope for Mailspring unless any open source release of Mailsync is expected.

[foundry376-bot]

This issue has been mentioned on Mailspring Community. There might be relevant details there:

https://community.getmailspring.com/t/protonmail-bridge-imap/485/1

[CodeMouse92]

Great news everyone! Mailsync is now open source! 🎉 🎉 🎉

Read more here: https://community.getmailspring.com/t/a-free-open-source-future-for-mailspring/484

We are in the process of migrating issues to Discourse, which can better facilitate discussion and discovery, and so GitHub Issues can focus on issues that are confirmed and slated for resolution in the near term. Learn more about the changes here.

As part of this, we've migrated this issue to Discourse:

https://community.getmailspring.com/t/protonmail-bridge-imap/485

Please consider joining that community and continuing the discussion there! Votes on the feature suggestion on Discourse will increase the likelihood we implement this.

@stellarsquall: if you join and reply to the issue, the moderators can make an effort to reassign your posts to you, so you get the credit for them.

We're closing and locking the issue here as part of this migration. Rest assured, this doesn't mean the issue is being discarded or ignored.

We hope to see you on Discourse soon!

-The Mailspring Team