Posting messages that raise errors will make the script loop forever

[shmulvad]

When posting the messages, I would get errors for a few of them such as

Caught HTTP exception sending message to channel: 400 Bad Request (error code: 50035): Invalid Form Body
In content: Must be 4000 or fewer in length.

Of course, trying to repost this will keep on giving the same error as the underlying message didn't change. This means the script will loop forever until manually stopped by the user.

I think you should provide an option like --max-retry-count , so if the same message keeps failing more than, say, 3 times, it will get skipped and perhaps logging all occurrences of this type to a log file. I had to manually change this part of the source code to not have the program get stuck in the same part every time I ran it.

[richfromm]

Apologies for taking so long to respond to this. I didn't see any of these open issues until today. See #22 (comment)

The underlying issue of exceeding discord limits is now covered in #29

I see the point you're making, but I'm not sure about the best way to handle failures in the general case. Part of the problem is that there are lots of things that could go wrong, and I don't have any easy way of anticipating these in advance, and/or testing for them. I'm kind of hoping that if more people use this I'll get more feedback about specific failure modes and can then explicitly handle them (like in this case, where I filed a new issue to deal with the actual failure).

So then what's the best thing to do for an unanticipated failure? The script in general does fail fast when its in the earlier stages, before any state has been changed in Discord. But once I start changing state in Discord, I was somewhat reluctant to give up for a general failure case in which I didn't know what was going wrong. Part of this is b/c the script is not idempotent, and doesn't have easy ways of restarting in the event of failures midway through. So I erred on the side of continuously retrying, forcing the user to make the call to give up. Which is noted in the discord_retry() docstring:

slack2discord/slack2discord/client.py

Line 321 in 2617f7e

This is not strictly the correct thing to do in all scenarios. But it's a lot more

Do you have any thoughts about differentiating retry-able failures vs. ones that will never succeed?

Is the Ctrl-C entry a problem? As in, if you press Ctrl-C does it really just stop, or do you get into some kind of loop where it restarts anyway and you have to press it repeatedly really fast, or something like that? Hopefully now that I have a way of knowing how to cause this failure, I can try a local test case to force it and see what the user experience is. Anticipating and testing failure modes has been difficult. IIRC, I tried to get the server to rate limit me with a HTTP 429 response and wasn't able to get it to happen.

Fwiw, I do have some thoughts about making the script a bit better about restarting in the event of failures, and at least moving somewhat closer to idempotency, if not being truly idempotent. For example, in the event of an incomplete run, output some identifier (partially based on msg timestamp?) that indicates how far it successfully got in the export, then provide some way of skipping to that point when re-running the script. But I haven't yet had the chance to explore any of this further.

At the very least, some notes in the README about unexpected failure cases and how to handle them might be worthwhile.

[richfromm]

I'm also curious what your preferred action is if giving up after a limited amount of retries. Do you expect the script to then just fail? Or to give up on that message and continue? I'm somewhat reluctant about the latter, given the difficulty of rerunning just part of a failure.

All of this is partly why I like in theory the idea of doing a dry-run first (although that won't manage to encounter all types of failures related to using the Discord API), but that's probably a more feasible bit of advice once I implement #26

[richfromm]

Logging failures to a file, like you suggested, and then leaving it up to the user to deal with those manually, is indeed an option.

[richfromm]

Do you have any other examples of situations that created errors that have not already been filed as issues?

While I believe there is room for potentially changing the behavior here, my preference and priority is to fix the issues causing the errors, with the goal being that this doesn't happen.

[shmulvad]

Do you have any other examples of situations that created errors that have not already been filed as issues?

I don't have any other examples. As you say, it might be good to leave it like it is if indeed everything (crossing fingers) is/will be handled now. Thanks for your work with all of this. I saw you responded to all my other opened issues as well.