Feature request: export chat logs

[sunnyolives]

A very useful feature. I find it strange it isn't already implemented, unless I am bad at searching.

Alternatives

• specify time range
• possibly, specify file format or formatting

What do you think?

[ara4n]

yup, i find myself wanting this repeatedly too. #2129 is close to it, but would be nice to just be able to say "download room as log".

[Half-Shot]

I was once toying with the idea of selecting some of the chat, and hitting some kind of key combo/ menu option to copy a IRC style log into the clipboard?

[ara4n]

This would also be cool!

[Nadrieril]

Any plans to do this ?

[ara4n]

as a p2 feature req this is basically stuck behind everything tagged as p1, which is around 200 issues right now. but given this is FOSS anyone is welcome to contribute it!

[clopez]

I find the search feature of matrix/riot pretty inefficient for an advanced user like me that is used to do greps (even using regexs) over simple text files.

So, being unable to extract simple plain text logs from my matrix conversations (both private or in rooms) is a serious regression for me, when compared with my previous experience with Jabber or IRC clients.

[ara4n]

https://gitlab.com/argit/matrix-recorder is a good workaround for this for now (and even does e2e if you desire, albeit storing it unencrypted). We could also build something like this into riot itself in future.

[MaskyS]

+1 for this feature request

[ghost]

I also need to inform that two my accounts at gmail and yahoo been locked and i have no access to them long time ago. I also worry that my citizen auth key was stored at that acc. Maybe anyone could help if you have my chats and so powerfullm

[scottAnselmo]

Being able to export IRC-like logs would be an extremely useful feature to have for quickly copying chat logs for reoccuring FOSS project meetings to archive somewhere. Glad to see it's now labelled p1 instead of p2.

[Valodim]

I miss this feature a lot and I'm tempted to give this a go. However the riot-web code base looks very daunting, especially with the weird separation of sdk and actual app. Can someone who has an overview of the code base give a rough plan of the relevant cutpoints where this would be implemented in riot-web?

[t3chguy]

@Valodim all the code would be in https://github.com/matrix-org/matrix-react-sdk/

[ilu33]

This feature is also required by GDPR. I think GDPR favours structured formats, json and the like, but I don't mind whether it's txt or anything else.
But besides the legal stuff it would be a really really useful feature. Copying the history manually is a p.i.t.a..
Also GDPR and other valid reasons convince server operators to delete older history (we are discussing something in between 3-6 month atm). So whatever is not copied gets lost.
So if @Valodim picks this up it would be great.

[t3chguy]

@ilu33 https://matrix.org/docs/projects/other/matrix-recorder.html
riot-web doesn't store the data so it doesn't have to be able to export it

[ilu33]

Nobody requires riot-web to store data. riot-web is just the user frontend and it pulls data from the hs all the time. If the user wants to store selected data the user frontend should provide a way to do so.

Using another tool is a crutch at best. How would the average enduser even install matrix-recorder? Not to mention https://gitlab.com/argit/matrix-recorder/issues/1 which makes the tool unusable for everybody who's in several high traffic rooms (which no sane person would want to archive). Also routinely archieving multi-user rooms without any reason or need to do so (just because the tool happens to be catch-all) defies every purpose of data protection. It might be allowed as long as you don't publish the data but it sounds immoral to me. I would not want to do that.

And regarding encryption: Matrix has already problems handling E2EE if more than one device is involved. It regularly breaks for no obvious reasons (issues are up here on github). I would not want to recommend using another device to anybody.

[t3chguy]

Yet on the other hand you run into much bigger limitations in webapps in the sizes of files you can generate before the browser kills you

[ilu33]

hm - limit the amount of lines? Maybe only export the stuff riot-web has already loaded into the browser window?

It seems that matrix-recorder has a similar problem (ORG.MATRIX.JSSDK_TIMEOUT)?

Note @t3chguy : I edited my previous comment.

[makedir]

Why is there nothing done since ticket open of Nov 22, 2016? The Riot client fronted should have a simple user-friendly basic chat export for admins, just copy chat from 1.1.2018 to 1.2.2018 in channel x to clipboard. It is also a stubborn and ridiculous excuse to say, client, doesn't need to have this feature because of limitations of size. Seriously? We are speaking of chat logs, not of file attachments. If you copy a chat log for a day or week its like 10-1000kb in size.

[Nadrieril]

@makedir It does indeed seem like a very useful feature, but please keep it civil. Developers have their own priorities. If you believe it is not very hard, why not give it a go ? Contributing to a super cool project like Riot is a fun experience !

[psaavedra]

JFYI, @thiblahute was developed a simple but useful command line for this:
https://gitlab.gnome.org/thiblahute/matrix-dl

In order to install matrix-dl without messing our system's Python setup, we will document how to install it using a virtual environment. The command virtualenv is provided by a Python package that enable the creation of these virtual environments. You can install virtualenv using you packaging system.

Run:

virtualenv -p python3 matrix
cd matrix
source bin/activate

Now, clone the code:

git clone https://gitlab.gnome.org/thiblahute/matrix-dl.git

And install the dependencies and the script itself in the virtual enviornment:

cd matrix-dl
python setup.py install

Usage:

The tool's usage instructions are these:

matrix-dl [-h] [--password PASSWORD] [--matrix-url MATRIX_URL]
          [--start-date START_DATE]
          username room

Download backlogs from Matrix as raw test

positional arguments:
  username
  room

optional arguments:
  -h, --help               show this help message and exit
  --password PASSWORD      Will be asked later if not provided
  --matrix-url MATRIX_URL
  --start-date START_DATE  format %d%m%Y

A couple examples:

Let's download the conversations from Example channel since the beginning of 2018:

matrix-dl --matrix-url https://matrix.example.com/ --start-date 01012018 \
  <fsurname> "Example" > example-2018.log

Then you will be asked for you password, and if there is no errors, the conversations will be dumped in the file example-2018.log with the format hh:mm:ss — @user: message

You can also dump conversation from unnamed rooms, such as personal conversation, you just need the room's internal ID. You can get this string in riot by clicking in the room's settings icon (the gear so far) and at the end of the settings, in the advanced section, there's the room's ID:

matrix-dl --matrix-url https://matrix.example.com/ --start-date 01012018 \
  <fsurname> \!i4BiDaYPkvfbcWdAgb:example.com > my-chat-2018.log

Remember to escape the symbol !, otherwise the shell may consider it an operator.

[Porkepix]

@jonathon-moore I advise you read the latest few comments/statuses.

It is being worked on. I just tasked the Design team with designing the UX for it.

I missed the fact it was already be worked on. Aside from one shot exports like it seems to be talked about here, does it also consider complete live logs, or would this need another issue, if this would ever be considered?

For a bit of context, on the irc side I rely heavily on grepping through logs, which I think no graphical interface would ever be as efficient as using correctly greps and similar tools.

[t3chguy]

Nope, this focus is for the webapp, the desktop app would inherit it for free. Webapps can't do such a thing so it is not being considered here.

[Porkepix]

Nope, this focus is for the webapp, the desktop app would inherit it for free. Webapps can't do such a thing so it is not being considered here.

What do you mean by "can't do such a thing", it's completely possible for the application to write in a plain text file what's being said in chans/conversations.

[t3chguy]

A webapp, as in a standalone Javascript app running in a supported web browser cannot write to your filesystem. It can (on explicit user interaction, e.g a download button) download a file but cannot just keep appending to a file.

https://developer.mozilla.org/en-US/docs/Web/API/File_System_Access_API / https://web.dev/file-system-access/ would allow for the latter but the user's consent would not persist between sessions plus the API is still a working draft. Forcing the user to consent each time they reload the app would not be a user-friendly experience.

https://web.dev/file-system-access/#stored-file-or-directory-handles-and-permissions
https://web.dev/file-system-access/#modifying-an-existing-file-or-directory

[neobrain]

https://developer.mozilla.org/en-US/docs/Web/API/File_System_Access_API / https://web.dev/file-system-access/ would allow for the latter but the user's consent would not persist between sessions plus the API is still a working draft.

Interesting, I was just about to propose this as an alternative here. Any chance this could be used for the Desktop app though? To me it's a very desirable feature to have a quickly searchable local backup without needing to export manually all the time, possibly more so than a dedicated export feature (though we ultimately might need both). Though I guess the FS API doesn't really get you anything over electron APIs, and then the biggest problem is to find someone to do the work...

Either way, thanks for pushing forward the work on this problem area!

[Porkepix]

That's right for the Web version, but can't Electron have better accesses to filesystem as it's a dedicated app?

[t3chguy]

Yes. The Electron app could do this.

...this focus is for the webapp, the desktop app would inherit it for free. Webapps can't do such a thing so it is not being considered here.

Nothing to prevent someone else from contributing an Element-Desktop specific variant which is able to keep an append only log. It is not the target of the Google Summer of Code project which will be closing out this issue.

[jaiwanth-v]

@neobrain I'll work on integrating that feature as well if I find time, but it'd be better to create a separate issue to track it.

[piegamesde]

@xelayz Please remain polite and constructive, and make sure you are adding value to the discussion. If you read the backlog, you can see plenty of messages describing their frustration about this feature missing, as well as viable workarounds.

[Sieboldianus]

btw. here is the link to the Pull Request for the export feature

[jakecoppinger]

I've been following this for a while and just exported using https://develop.element.io/ - absolutely amazing stuff, thanks so much for your hard work!

[xelayz]

I've been following this for a while and just exported using https://develop.element.io/ - absolutely amazing stuff, thanks so much for your hard work!

Wow, thats perfect. Is this a new developer feature? Because no one told us before this is possible over the developer site.

[t3chguy]

develop.element.io has stuff which will be included in the next release, so this will be in that next release.

[andyortlieb]

develop.element.io has stuff which will be included in the next release, so this will be in that next release.

Will those features come out to the next Android release as well?

[SimonBrandner]

Will those features come out to the next Android release as well?

No, element-android uses a different codebase, for an element-android issue see https://github.com/vector-im/element-android/issues

[Felix-Pedro]

hey, is there a way for admins to block users from doing this?

[piegamesde]

@Felix-Pedro why should there be one? This smells like poor threat modelling to me honestly, but I'd be curious about your motivation for this.

[Felix-Pedro]

@piegamesde is not a formal feature request or anything like that, I was playing with the app trying to understand how it works, I'm new to matrix so I was wondering if there would be any privacy concerns about exporting unencrypted copies of conversations, an option to encrypt would probably be nice too or like if the chat is encrypted the app could advise or admins could enforce encrypted exports or admin notification of exports, my primary source of concern was hypothetical, I was trying to list good usages for people that need some security on their conversations and thought about persecuted journalists, organizations, etc . And the risk of such export in this situation. But I could also be completely wrong about this.

[t3chguy]

@Felix-Pedro in an open protocol you can never enforce any such block, someone could always modify their client to remove the block. The export just uses the same API as a user scrolling back through a room. A user could also just video record themselves scrolling all the way up as a form of export.

[Felix-Pedro]

@t3chguy I thought of video issue too, but not the client modify, I understand that if one is decided to do so, and it has access to the chat, they will get the information extracted, but like banking apps that make your screenshot black when at some parts of the app, it creates a way to make it harder for unwilling leaks, just an option to encrypt the export or an advise from the app itself that this is an encrypted chat and would be better to encrypt the export too, would probably be enough, i' hadn't thought of this possibility before, although it's not a huge urgent issue I still think the option to encrypt and advice to do so would be a good feature. But I see that enforcing it does not prevent it from happening, and now I see too it does not make sense on the platform.

[t3chguy]

but like banking apps that make your screenshot black when at some parts of the app

Have you seen any banking websites which do this? After all this is a webapp and websites cannot blank your screen. Plus given its open source someone could again just modify and remove whatever hurdles you choose to add, or simply use a different Matrix client given there are hundreds.

[Felix-Pedro]

Ok i'm not asking for enforcing it, I'm asking for an encryption option and an advice to do so, not enforce anything, also for the non technical user changing app codebase to remove a warning is not a simple nor worth task.

[t3chguy]

also for the non technical user changing app codebase to remove a warning is not a simple nor worth task.

No they'll just google "How to get around Element export chat blocked" and someone with the ability will have a fork/alternative available to them. Security via obscurity is not something we're interested in.

[Felix-Pedro]

For god sake how many times I have to say that't is not a block, it's just a warning message, and check box yes encrypt or no do not encrypt.

[remram44]

If you have a feature request, maybe open a new ticket?