Deal with discord character limit on message text

[richfromm]

#25 complains about a situation in which a failure that is never going to pass gets stuck in a retry loop forever.

I'll address the retry situation in that issue, but the more pressing concern IMHO is that discord has character limits that I wasn't aware of, and that I think need to be dealt with. (Via either truncation, splitting up, or perhaps some combination of the two.)

In the case reported, the limit was 4000. Most sources I've seen say there's a 2000 character limit per message, but I think you can get higher limits if you pay? But sadly it's not just message contents that might be the problem, there could be issues with embeds as well.

Some refs:
https://www.itgeared.com/what-is-the-character-limit-on-discord/
https://discord.com/developers/docs/topics/opcodes-and-status-codes
https://www.google.com/search?q=discord+maximum+message+length&oq=discord+maximum+mes&aqs=chrome.0.0i512j69i57j0i22i30j0i390l4.4002j0j7&sourceid=chrome&ie=UTF-8
https://www.integromat.com/en/help/how-to-split-and-post-several-messages-without-exceeding-a-certain-character-limit-e-g-discord
https://discord.com/developers/docs/resources/webhook#execute-webhook-jsonform-params
https://www.reddit.com/r/discordapp/comments/lfawsu/why_is_the_discord_message_character_limit_is_2000/

Attn: @shmulvad

[richfromm]

I'm going to file embed-related limits as a new issue, and limit this issue to the limit on message text.

I am able to reproduce this, a message in slack of 1000 characters was posted successfully, but a message of 2000 characters was not:

# ...
2023-02-07 10:39:01 INFO     slack2discord.client Message posted: 1675747843.005939
2023-02-07 10:39:01 INFO     slack2discord.client Message posted: 1675747855.982359
2023-02-07 10:39:01 INFO     slack2discord.client Message posted: 1675747858.401159
2023-02-07 10:39:01 WARNING  slack2discord.client Caught HTTP exception sending message to channel: 400 Bad Request (error code: 50035): Invalid Form Body
In content: Must be 2000 or fewer in length.
2023-02-07 10:39:01 INFO     slack2discord.client Will retry #1 after 1 seconds, press Ctrl-C to abort
2023-02-07 10:39:03 WARNING  slack2discord.client Caught HTTP exception sending message to channel: 400 Bad Request (error code: 50035): Invalid Form Body
In content: Must be 2000 or fewer in length.
2023-02-07 10:39:03 INFO     slack2discord.client Will retry #2 after 5 seconds, press Ctrl-C to abort
2023-02-07 10:39:08 WARNING  slack2discord.client Caught HTTP exception sending message to channel: 400 Bad Request (error code: 50035): Invalid Form Body
In content: Must be 2000 or fewer in length.
2023-02-07 10:39:08 INFO     slack2discord.client Will retry #3 after 30 seconds, press Ctrl-C to abort
2023-02-07 10:39:38 WARNING  slack2discord.client Caught HTTP exception sending message to channel: 400 Bad Request (error code: 50035): Invalid Form Body
In content: Must be 2000 or fewer in length.
2023-02-07 10:39:38 INFO     slack2discord.client Will retry #4 after 30 seconds, press Ctrl-C to abort
�
^C
2023-02-07 10:39:44 INFO     slack2discord Discord import from Slack export complete (may or may not have been successful)

Note that message length in Discord is longer than in Slack, since we prepend a pseudo-header within the text to indicate the timestamp and original author. e.g.:

                   1675747858.401159: (ParsedMessage(text='`2023-02-06 21:30:58` **rich** next 2000', links=[], files=[]),
                                       None),
                   1675747871.124789: (ParsedMessage(text='`2023-02-06 21:31:11` **rich**
012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678
012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678
012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678
012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678
012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678
012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678
012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678
012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678
012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678
01234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567

012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678
012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678
012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678
012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678
012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678
012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678
012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678
012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678
012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678
012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678', links=[], files=[]),
                                       None),

We can truncate above the limit, or split up messages. Truncating is obviously easier, but I think splitting up is the right answer.

But the implementation might be a little tricky, esp. if you account for threading. My proposed methodology is as follows:

• If the message is not related to a thread, just expand it into multiple messages, all also not related to a thread, but in the main channel conversation.
• If the message is the root of a thread, then expand it into multiple messages, and all of the new messages are inserted into the thread as the very first messages, before any existing messages that make up the actual thread.
• If the message is within a thread, then expand it into multiple messages, inserting the new messages within the existing thread immediately after the original message.

Note that there's already a special case where messages are in a thread but we can't find the root, and we deal with that by creating a fake message for the root of the new thread. Need to make sure that any of this doesn't interfere with that behavior, although I don't think that should be a problem.

[richfromm]

Issue narrowed in scope to be just message text limit, embed related limits are now covered in #30

[waocats]

I wrote some code on my end to fix this during my migration; my apologies for not getting around to filing an issue for it. It was a bit of a stopgap measure though, so I'll work on cleaning it up and pushing when I have more time later this week.

[richfromm]

Okay, I've also got some ideas on how to handle this in a not too hacky way. Feel free to work on it further, or to just post what you have if you want me to have a look at its current state.

I've got enough other issues to work on that I can put this on hold a little bit and address some of the other ones first.

[shmulvad]

In the case reported, the limit was 4000. Most sources I've seen say there's a 2000 character limit per message, but I think you can get higher limits if you pay?

Not sure how much the above matters to you, but I am not using the payed version of Discord. However, I think I also had a run where I got the error message with the number 2000 being reported and not 4000. It might have been an edge case with an embedding or thread.

So the safest thing is probably to use 2000 as the limit.

[waocats]

Okay, I've also got some ideas on how to handle this in a not too hacky way. Feel free to work on it further, or to just post what you have if you want me to have a look at its current state.

I've got enough other issues to work on that I can put this on hold a little bit and address some of the other ones first.

Feel free to work on the other issues when I get around to this. My last semester of university started not too long ago, so I unfortunately don't have as much free time as I used to 😓

That being said, I'll look into it and see what I can do today/this weekend.

[waocats]

Sorry for not getting to this sooner; it's been very busy recently. I can try to look into it this weekend instead, if I have some extra time.

[richfromm]

Don't sweat it either way. I already have a vision for how I was thinking of doing it, and an order of issues I was going to get to, so I'll probably get this done pretty soon, if you have other things keeping you busy. And now that I've picked this up again and refreshed my memory after having put it aside for a few months, it's relatively easy going.

I very much welcome discussion of features and approaches, and specific PR review. Actual PRs by others might be useful too, but I'm also fine with just or mostly the former.

[oubiwann]

I hit this today when trying to migrate an open source project's Slack to Discord ... definitely 2000 chars:

2023-03-17 13:54:59 INFO     slack2discord.client Will retry #22 after 30 seconds, press Ctrl-C to abort
2023-03-17 13:55:29 WARNING  slack2discord.client Caught HTTP exception sending message to channel: 400 Bad Request (error code: 50035): Invalid Form Body
In content: Must be 2000 or fewer in length.