Support for Mattermost v4

[Swarthon]

As Mattermost has developped a v4 which is incompatible with the previous versions of the API, I would like to know if there is any kind of project to support this new version.

[AlexeyGusev]

plus one vote. Any plans to add support for v4?

[ylluminate]

Are there any differences on v5 (and 5.1) that need to be accounted for?

[EionRobb]

Oh damn, we still didn't catch up to v4 and v5 is already happening?

I'll make this my priority over the next week to get sorted.

I'll have a look at the docs and see what changes there are between v4 and v5, but I plan to write it all in such a way to keep v3 -> v5 compatibility (I'm still connected to a server that's on v3 and doesn't look like it's going to be upgrading any time soon)

[p5n]

@ylluminate mattermost-server dropped API v3 support since version 5.0, so you cannot use this plugin until API v4 supported.

[ltning]

@EionRobb Anywhere/any way we can throw in a bit of .. ahem .. motivational support to get this done sooner rather than later? :)

[EionRobb]

My wife just had her baby early. There may be some delay getting back to this, sorry. :)

[jaroslawp]

Hi all !
I finally found some time to start looking at the initial v4 API implementation (especially that my server dropped v3 now ...). Here is 1st try: https://github.com/EionRobb/purple-mattermost/tree/v4api_try1
(be sure to READ the notes about what to expect so far ...)

@EionRobb :
Keeping v3 compatibility will need a lot of 'ifs' in the code it seems:

• api endpoints changed
• api methods changed (adding HTTP/1.1 PUT)
• and also api responses changed too
  maybe just freezing current release as last one working with v3 and develop further features with v4 only would be enough ? .. *I think all MM server releases since 13 months are v4 api capable)
  Let me know ...

[jaroslawp]

Call for testers: the v3 -> v4 API porting should be complete now, please try: https://github.com/EionRobb/purple-mattermost/tree/v4api_try1 (anybody using pidgin/libpurple 3 ? .. I only tired it with 2)

[AlexeyGusev]

@jaroslawp Thank you for your time and effort.

Using libpurple 2.10.12-0ubuntu5.2. Following commands:

cd purple-mattermost
git pull
git checkout v4api_try1
make 
make install

produced a valid .so file without any problems. Here comes first set of test results:

1. In pidgin, immediately after login to mattermost account (using gitlab cookie) got this:
   
   Hitting close button dismisses the dialog. No obvious glitches in relation to that.

2. Pidgin shows a mix of end users (those that are under DIRECT MESSAGES category in mattermost client) and subscribed channels (private and public) in one list, A-Z sorted. My proposal is to show channels/rooms in the end of the list or in Chat subsection (however, channels end up in a Chat subsection when they are "added" from the room list. Don't know which way is better, maybe subscribed channels should be "added" automatically).

3. Conversation with end-users works as expected. Re-opening the window (double-click on user in a list) shows previous chat history.

4. Room list button in Joining a chat window produces correct list of channels, type (private/open) is also correct. Can add a channel to a chat subsection or join a chat.

5. It is possible to write to a subscribed channel and see responses while the window is open. Closing the channel (chat) window and reopening it again will end up in lost history — this is obviously not as desired, contents should be automatically filled in when the room is open (either full history or some period/size, perhaps adjustable in settings).

Waiting for try2...

[p5n]

Works for me with private messages in spectrum2 as jabber transport, but #60 still occurs.

[jaroslawp]

@AlexeyGusev : Thanks for testing !

Ad1: OK, please checkout & compile again - I made that error silent for now (it seems that v5 server has no endpoint api path defined for non-defined user preferences: that 'error' is not affecting anything, and will be gone when user 'deletes' a direct / group channel from Web UI)

Ad2: This is rather feature request independent of v3 -> v4 API migration, please open a separate one.

Ad5: This is a bug also already present in v3 API I believe - please open a separate bug report.

@p5n : Thanks for testing ! yes .. this is only v3->v4 API update ... for #60 (and others ... I will try to find some time after summer holidays ..)

[p5n]

Previously I used adapted for spectrum2 and xmpp own fork: https://github.com/p5n/purple-mattermost

It works slightly better with xmpp, but I did not port it to v4 yet. This fork has fixed #60, but it has also many changes regarding convenient use in XMPP.

[jin-eld]

Tried just now using pidgin, seems to work so far, thank you very much for updating the plugin!

[jaroslawp]

@jin-eld : thanks for testing, latest commit also includes chat history rewrite (still to be polished a little..)

[jin-eld]

@jaroslawp What does it do? Will it fetch the messages from the server that were written to me while I was offline? I was missing that in the old v3-API plugin version :)

Btw, when compiling there's a little warning:

libmattermost.c: In function ‘mm_find_channel_approximate_view_time’:
libmattermost.c:1521:16: warning: initialization makes integer from pointer without a cast [-Wint-conversion]
  gint64 then = NULL;
                ^~~~

Does not look like anything serious, but I thought I'd still mention it.

[jin-eld]

Last commit coredumps :( I'll see if I can get a backtrace.

[jin-eld]

I hope this helps, its enough to just let it run for a while.

#0  0x00007ffff2ff52a0 in g_str_hash () at /lib64/libglib-2.0.so.0
#1  0x00007fffe45fae23 in mm_mark_room_messages_read_timeout (userdata=0x555556055560) at libmattermost.c:4551
#2  0x00007ffff30065dd in g_timeout_dispatch () at /lib64/libglib-2.0.so.0
#3  0x00007ffff3005b77 in g_main_context_dispatch () at /lib64/libglib-2.0.so.0
#4  0x00007ffff3005f20 in g_main_context_iterate.isra ()
    at /lib64/libglib-2.0.so.0
#5  0x00007ffff3006232 in g_main_loop_run () at /lib64/libglib-2.0.so.0
#6  0x00007ffff6b19827 in gtk_main () at /lib64/libgtk-x11-2.0.so.0
#7  0x000055555558f9d5 in main ()

I am on commit d1ec531

[jaroslawp]

@jin-eld : many thanks for testing, the latest commit possibly fixes these.
Just to make it clear: the code quality on this branch is 'rapid development/experimental' style :-) .. probably will segfault in other places too (not talking about some memory leaks ...).
Yes: the idea is to get the history working and get it somehow integrated smoothly with pidgin UI. What is in there now works .. let's say 'kind of' ... some more code digging in next week should happen...

[jin-eld]

@jaroslawp : if you need more info in places which you are unsure about, just add debug logs and I'll start pidgin in a terminal with debug output.

I more or less have no choice, our mattermost server at work has been updated and since I find the browser UI to be terrible and unpractical I am really happy there is at least something that is working with pidgin again :) I understand that its work in progress, that's fine, most importantly - its already usable, so thank you for coding it, keep hacking!

I'll see that I stay up to date with your branch, so I'll let you know if something crashes or does not work as expected ;) That being said, "search for users" does not find anyone, I assume you did not have time for it yet?

[jaroslawp]

You can run pidgin with --debug option to capture logs (be aware these will contain lots of personal info..)

As for search for users: interesting - that works for me, please check with pidgin --debug, search for users and look for lines in the log alike:

 mattermost: Fetching url https://mattermost.server/api/v4/users/search
mattermost: With postdata {"term":"a","allow_inactive":true,"without_team":true}

then the reply (note these can be interleaved with other replies in the log):

util: Response headers: 'HTTP/1.0 200 OK
Content-Type: application/json
X-Request-Id: q1j36m5m17drdmq6cpbiku3h8c
X-Version-Id: 5.1.0.XXX
Date: Mon, 06 Aug 2018 18:57:42 GMT

'
(20:57:27) mattermost: Got response: [{"id":"m7jizakp5pr99pru4to8qxx8tw","create_at":1482245695465,"update_at":1482245695465,"delete_at":0,"username":"00c9afb7642c2e80f9d5","auth_data":"","auth_service":"","email":"xx.zz@yy","nickname":"","first_name"xx","last_name":"zz", ...

[jin-eld]

Thats weird, my log looks like yours, except that I get [] as a response... this used to work in v3, but then again - our server was updated, maybe its a server side problem.

(22:27:58) mattermost: Fetching url https://myserver.org/api/v4/users/search
(22:27:58) mattermost: With postdata {"term":"usernickname","allow_inactive":true,"without_team":true}
...

(22:27:59) util: Response headers: 'HTTP/1.1 200 OK
Server: nginx/1.12.2
Date: Mon, 06 Aug 2018 19:11:34 GMT
Content-Type: application/json
Content-Length: 2
Connection: close
X-Request-Id: XXX
X-Version-Id: 5.0.0.5.0.2.YYY.false

'
(22:27:59) util: parsed 2
(22:27:59) mattermost: Got response: []

[jin-eld]

I noticed that the history part does not seem to work properly. It does not correctly download offline messages from the server and now I had a case where my current chat was suddenly reset to something older (i.e. messages in the chat window changed, jumping back in time).

This may be related to logging in to the server via a browser in parallel (I do that once a day to check if I missed any offline messages).

Edit: it seems that I am also missing messages from time to time. A new tab chat popped up, but displayed old history. I did not have a second browser session to the server, was connected only via pidgin. Then I doublel checked on the server via browser chat, and saw that I was actually supposed to have a new message in the chat that popped up, however in pidgin this message was missing.

[jaroslawp]

Many thanks for testing !
This commit : cfdd873 should fix the history processing - since I could not get consistent results trying to synchronize history with server ... (not sure why, possible I miss something in API calls sequence ...) this time I implemented our own history timekeeping (so we should get history as long as this client have not read it).

Lets test if that works better now. There are still bugs described in README and there may be some crashes (possibly) and some memory leaks (likely).

[jin-eld]

The behaviour got better, but I had one crash, unfortunately no trace. Just had it running, did not do anything special.

The second crash happened when I logged in via a browser in parallel to my running pidgin session:

#0  0x00007ffff2ff52a0 in g_str_hash () at /lib64/libglib-2.0.so.0
#1  0x00007ffff2ff46f4 in g_hash_table_lookup () at /lib64/libglib-2.0.so.0
#2  0x00007fffe45f9b68 in mm_get_history_of_room (ma=0x5555561d0450, channel=, since=-1) at libmattermost.c:4362
#3  0x00007fffe45fb5b5 in mm_get_users_by_ids_response (ma=0x5555561d0450, node=, user_data=0x55555613b300) at libmattermost.c:2080
#4  0x00007fffe45f20c9 in mm_response_callback (http_conn=, user_data=0x5555560d6940, url_text=, len=, error_message=0x0) at libmattermost.c:1146
#5  0x00007ffff4e0c9f5 in url_fetch_recv_cb () at /lib64/libpurple.so.0
#6  0x00005555555c94be in pidgin_io_invoke ()
#7  0x00007ffff3005b77 in g_main_context_dispatch ()
    at /lib64/libglib-2.0.so.0
#8  0x00007ffff3005f20 in g_main_context_iterate.isra ()
    at /lib64/libglib-2.0.so.0
#9  0x00007ffff3006232 in g_main_loop_run () at /lib64/libglib-2.0.so.0
#10 0x00007ffff6b19827 in gtk_main () at /lib64/libgtk-x11-2.0.so.0
#11 0x000055555558f9d5 in main ()

This is nicely reproducible btw.

[jaroslawp]

Thanks !, last commit (76f5faf) should fix most of these problems / crashes.

It seems that the plugin is working now for v4 API with same functionality it had for v3 API, there are some memory alloc/dealloc to review plus more debugging/testing needed.

I will be away from keyboard for next two weeks .. so further followup on this .. only in September ..

[jin-eld]

Great, thank you! Seems to work so far, the crash is gone.

Let's see if I come up with more traces till September ;)

[robert-scheck]

Being forced to use Mattermost (rather XMPP in the past) at work, I tried 76f5faf: Works except for rooms/channels; on the first connect I see e.g. 2480 persons in a room/channel using libpurple while the room/channel has actually < 20 people. It heavily varies for each room/channel, likely related to the amount of written messages? However when closing the channel in Pidgin and re-opening it, it looks normal as it seems?!

Oh and when trying to use this libpurple plugin via https://github.com/bitlbee/bitlbee, a failed connect to the Mattermost server (e.g. wrong password, expired/hijacked GitLab authentication cookie) hangs up the whole BitlBee somehow (until BitlBee process is killed using SIGKILL):

<@root> eionrobb-mattermost - Login error: Connection error: Error reading from <server>: Input/output error.
-!- ** (process:8516): CRITICAL **: mm_update_cookies: assertion `headers != NULL' failed
-!- (process:8516): GLib-CRITICAL **: g_strstr_len: assertion `haystack != NULL' failed
-!- (process:8516): GLib-CRITICAL **: g_regex_unref: assertion `regex != NULL' failed
-!- (process:8516): GLib-CRITICAL **: g_regex_unref: assertion `regex != NULL' failed
-!- (process:8516): GLib-WARNING **: giounix.c:411Error while getting flags for FD: Bad file descriptor (9)
-!- ** (process:8516): CRITICAL **: mm_update_cookies: assertion `headers != NULL' failed
-!- (process:8516): GLib-CRITICAL **: g_strstr_len: assertion `haystack != NULL' failed
-!- (process:8516): GLib-CRITICAL **: g_regex_unref: assertion `regex != NULL' failed
-!- (process:8516): GLib-CRITICAL **: g_regex_unref: assertion `regex != NULL' failed
-!- Irssi: warning Die Verbindung wurde vom Kommunikationspartner zurückgesetzt
-!- ** (process:8516): CRITICAL **: mm_update_cookies: assertion `headers != NULL' failed
-!- (process:8516): GLib-CRITICAL **: g_strstr_len: assertion `haystack != NULL' failed
-!- (process:8516): GLib-CRITICAL **: g_regex_unref: assertion `regex != NULL' failed
-!- (process:8516): GLib-CRITICAL **: g_regex_unref: assertion `regex != NULL' failed
-!- ** (process:8516): CRITICAL **: mm_update_cookies: assertion `headers != NULL' failed
-!- (process:8516): GLib-CRITICAL **: g_strstr_len: assertion `haystack != NULL' failed
-!- (process:8516): GLib-CRITICAL **: g_regex_unref: assertion `regex != NULL' failed
-!- (process:8516): GLib-CRITICAL **: g_regex_unref: assertion `regex != NULL' failed

[AlexeyGusev]

on the first connect I see e.g. 2480 persons in a room/channel using libpurple while the room/channel has actually < 20 people

similar behaviour here, but every user gets multiplied by random number in the list of room users: in one room, you could see 6 entries or User A, 6 entries of user B and so on for all given room users; in another room, you could see 100 entries of User A, 100 entries of user B etc.

[jin-eld]

Interestingly enough the above does not happen for me (using Pidgin), but then again I have only one room with 6 people where I'm in.

[robert-scheck]

Interestingly enough the above does not happen for me (using Pidgin), but then again I have only one room with 6 people where I'm in.

I think, I was able to track it down: I only see these zillion person duplicates in a room/channel when I was added on server-side to a room/channel. If I remove the room/channel from the buddy list in Pidgin explicitly and join the room myself using Pidgin, it looks normal.

And if purple-mattermost would work with BitlBee (rather just hanging it up like mentioned above), it really would be great :)

[jaroslawp]

User list duplicates should be fixed now.

As for history: it .. may .. be (mostly) fixed now
(my MM server seems to return whole history since requested time despite docs saying it should return it as paged ? ... so please check with yours .. the difference may show if there are more than 60 messages in history only).

If you see duplicates in history parts: these may come from client logs.

History ordering is slightly incorrect:

• Edited messages (original of edited messages shows with wrong timestamp)
• Pasted images (correct timestamp but shown usually at the end of history due to how messages are processed .. image requires additonal async call to the server so before it is processed other history messages are already displayed)

[AlexeyGusev]

ver 0e126dd

• no user duplicates so far.
• private (buddy) chat history is displayed properly.
• room history is still a matter of random luck, sometimes it appears, sometimes not, even for the same chat. What seems to always malfunction is disappearance of history from the chat windows once it is closed and then reopened in the same pidgin session.

[AlexeyGusev]

(12:21:31) plugins: /usr/lib/purple-2/libjabber.so is not usable because the 'purple_init_plugin' symbol could not be found.  Does the plugin call the PURPLE_INIT_PLUGIN() macro?
(12:21:31) plugins: probing /usr/lib/purple-2/liboscar.so
(12:21:31) plugins: /usr/lib/purple-2/liboscar.so is not usable because the 'purple_init_plugin' symbol could not be found.  Does the plugin call the PURPLE_INIT_PLUGIN() macro?

happens during pidgin startup, maybe has some value, not just noise

[jin-eld]

Looks good so far, no duplicates for me either, thank you!

However I see another small issue:

I disabled notifications for a channel, but I still get messages from there in pidgin. It's some persistent channel that I can't get rid of, but disabling notificatoins was possible in the mattermost web UI.

Is there something missing in the plugin to supress messages when channel notifications are disabled or is there any other way to ignore it?

[jin-eld]

There's one more issue that I noted, when a new user joined a channel, his nickname appeared empty in the nicknames list on the right hand side. In the chat the message "user xx joined the channel" was correct though.

[jaroslawp]

@jin-eld: thanks for reporting - can you try this branch: https://github.com/EionRobb/purple-mattermost/tree/cleanup-1 (please move away ~/.purple/blist.xml again). I've made some code cleanup for setting up aliases for users/chats and this may fix it.