Snikket doesn't store offline messages

[Hund]

I'm not sure how to word the title for this. I'm self-hosting an instance with Snikket, and I have an issue with Snikket not storing offline messages. So, when I login to my client (Gajim for Linux and Snikket for Android), I don't see any messages that was sent to me while I was offline.

I have made sure that this is not an issue with Gajim. One of the them helped me debug the issue with their client just to make sure it's not a client side issue.

This is my docker-compose.yml config:

version: "3.3"

services:
  snikket_proxy:
    container_name: snikket-proxy
    image: snikket/snikket-web-proxy:stable
    env_file: snikket.conf
    network_mode: host
    volumes:
      - snikket_data:/snikket
      - acme_challenges:/var/www/html/.well-known/acme-challenge
    restart: "unless-stopped"
  snikket_certs:
    container_name: snikket-certs
    image: snikket/snikket-cert-manager:stable
    network_mode: host
    env_file: snikket.conf
    volumes:
      - snikket_data:/snikket
      - acme_challenges:/var/www/.well-known/acme-challenge
    restart: "unless-stopped"
  snikket_portal:
    container_name: snikket-portal
    image: snikket/snikket-web-portal:stable
    network_mode: host
    env_file: snikket.conf
    restart: "unless-stopped"

  snikket_server:
    container_name: snikket
    image: snikket/snikket-server:beta
    network_mode: host
    volumes:
      - snikket_data:/snikket
    env_file: snikket.conf
    restart: "unless-stopped"

volumes:
  acme_challenges:
  snikket_data:

And this is my snikket.conf config:

# The primary domain of your Snikket instance
SNIKKET_DOMAIN=<DOMAIN>

# An email address where the admin can be contacted
# (also used to register your Let's Encrypt account to obtain certificates)
SNIKKET_ADMIN_EMAIL=<E-MAIL>

# Hund addes this for reverse proxy to work
SNIKKET_TWEAK_HTTP_PORT=5080
SNIKKET_TWEAK_HTTPS_PORT=5443

SNIKKET_RETENTION_DAYS=14

I have no idea how to debug this issue.

[Hund]

This is quite a severe bug if you ask me. :(

[mwild1]

In Snikket Android, from the main screen:

1. Open the menu (three dots, top right corner)
2. Tap 'Manage accounts'
3. Tap on your account which is not receiving offline messages
4. Open the menu (three dots, top right corner)
5. Tap 'Archiving preferences'
6. Ensure 'Always' is selected
7. Tap 'OK'

If 'Always' was already selected and these steps do not resolve your issue, let me know.

[Hund]

Always is already selected. :)

[Hund]

I would highly appreciate some help with this. Not receiving any messages while being offline is a major inconvenience for us. My instance have about 30 users.

[mwild1]

Really sorry about the inconvenience. Snikket is supposed to "just work", and I understand it's frustrating when stuff doesn't work.

Just for the record, I have been helping (in the chat) and we made some progress thanks to the information you were able to provide. I'm sorry to say that despite the couple of hours I have already spent just on looking into this issue, I'm still unable to say what is causing your problem.

Nobody else has reported this issue, and I've not seen anything like it before, so I'm fairly confident that it's specific to your setup somehow. We just need to figure out what's special. I have lots of other open tasks, and there is only so much time I can spend debugging a one-person issue. However, I am doing my best :)

What we've verified so far in debugging with you:

• The problem is not due to lack of disk space
• The problem is not due to corrupt files in storage
• The problem is not due to a read-only mount
• Prosody logs that it is unable to archive stanzas (confirming there is a problem)
• There seem to be log lines strangely missing from your debug logs (including, importantly, the one that would tell us why archival failed)

For the next steps, it would be good to get past that last problem. If we can find the missing log lines (or why they are missing) we should be able to get closer to solving the main mystery.

Previously I instructed you to read the logs using docker compose. If you can, let's try a different method.

1. cd /etc/snikket
2. docker-compose exec snikket_server prosodyctl shell watch log | tee -a /tmp/snikket-debug.log (this will keep running - keep the terminal open and let it run while you perform the next step)
3. Arrange for one user to send a message to another user who is affected by the offline message issue
4. Back in the terminal, press Ctrl+C to stop the logging (you may need to press it multiple times before you get back to the prompt)
5. Send the /tmp/snikket-debug.log to me via email like you did with the previous log file, and I will analyze it for problems.

Fingers crossed that it will provide the information we need!

[Hund]

Done. I sent the message "TEST" from the user johan to the user billy.

[mwild1]

For future reference, we had some further exchanges via chat/email. The captured logs included the missing log lines we were looking for. They indicate that our earlier conclusion about no corruption was false. The log contains errors such as:

Failed to load archive storage ('/snikket/prosody/domain/archive/user.list:38212: unexpected symbol') for user: user@domain

This kind of corruption has been known to happen in some cases after a full disk or power failure event.

The recommended solution in this case was to simply remove the files, which would clear the errors and allow Prosody to recreate the files when new messages come in.

In the long term, we'll likely be moving to SQLite (per #196) which will hopefully be more resilient to whatever caused this.