[Bug]: 🪲 Server side encryption does not encrypt files with S3 primary storage, related error messages on encrypt-all and scan:legacy-format 😢 #41992

asheroto · 2023-12-03T16:36:52Z

⚠️ This issue respects the following points: ⚠️

This is a bug, not a question or a configuration/webserver/proxy issue.
This issue is not already reported on Github OR Nextcloud Community Forum (I've searched it).
Nextcloud Server is up to date. See Maintenance and Release Schedule for supported versions.
I agree to follow Nextcloud's Code of Conduct.

Bug description

Preface

This issue is well reported and discussed around GitHub and the Nextcloud Community forum, but the closest Issue is #33371 but @szaimen said to open a new bug report for ongoing issues which is why I'm creating a new issue for this. I'm hoping this issue gets solved. It seems like Nextcloud has a lot of issues with encryption, especially with S3. Many of these issues were reported months or years ago. 😊

Maybe most people do not use S3 as their primary storage, I'm not sure? S3 is an inexpensive way to store lots of data, especially with providers like Wasabi because it's only $6/TB/month. Hoping that someone takes ownership of this situation, even if that just means preventing new installs from using it or adding something to the manual. 😊

Remaining optimistic!

Background

Using Wasabi as an S3-compataible provider for primary storage in Nextcloud.

After upgrading from Nextcloud 26 or 27, I started receiving errors related to encryption (#8349) and thought I was able to solve it by disabling encryption and re-enabling it, but unfortunately most files remained unencrypted. After hours of troubleshooting, I decided to just start over from scratch.

So I installed a clean Nextcloud instance with a new database, new Redis, everything. But as soon as I set up S3 and enabled server side encryption, I ran into trouble. Files will sync and upload to the S3 provider but are not encrypted, unless they are very small. For example, text files will upload and encrypt, but even an 8 MB zip file will not encrypt. Same result in web client and in Windows client.

Because there are many similar issues, and even with a fresh install issues exists, it seems that there are still some unresolved issues within the codebase when it comes to this type of setup: S3 + server side encryption.

Troubleshooting

After performing the Steps to reproduce and then experiencing the issue, I troubleshooted by running these commands, testing with file uploads after each command:

The logs only appear when running or after running the occ commands. If I do not run the commands, I do not see anything in the log files. The uploads complete as expected without any logging, but the files are not encrypted.

Steps I have tried to resolve:

Decrypt all files: sudo -u www-data php occ encryption:decrypt-all
Verify encryption disabled: sudo -u www-data php occ encryption:status
Disable encryption: sudo -u www-data php occ encryption:disable
Scan all files to ensure integrity: sudo -u www-data php occ files:scan --all
Uncheck the Encrypt the home storage option in Settings > Security
Wait a minute
Enable encryption: sudo -u www-data php occ encryption:disable
Check the Encrypt the home storage option in Settings > Security
Wait a minute
Ensure encryption is enabled: sudo -u www-data php occ encryption:status
Scan key storage format: occ encryption:scan:legacy-format
Migrate key storage format for good measure: sudo -u www-data php occ encryption:migrate-key-storage-format
Encrypt all files: sudo -u www-data php occ encryption:encrypt-all

After running that, all of the files are encrypted.... but no new files become encrypted.

Similar issues

Related issues that I have read, implementing fixes and pieces from each:
Server Side Encryption Does Not Encrypt Files When Using S3-Compatible Primary Storage #33371
Encryption not working with S3 object storage as primary storage #11826
Server-side encryption is incompatible with using an object store as a primary data store #22077
Nextcloud update cause encryption files not opened anymore or being encrypted. #2206
Enabled encryption and disabled it again and now it throws me an error in the interface and the logs

Similar family of issues that originally lead me to reinstall Nextcloud:
Encrypt:scan:legacy-format finds files which doesn’t have a proper header in files_version and files_trashbin
Invalid private key for encryption app. Please update your private key password in your personal settings to recover access to your encrypted files #8546
Nextcloud Encryption breaks with OpenSSL 3.x due to legacy RC4 usage #32003
Fix encrypted version to 0 when finding unencrypted file #28373
occ encryption:scan:legacy-format
Nextcloud 20 The old server-side-encryption format is enabled #22478

Many people are simply disabling the "Default Encryption Module" but that is not a solution because it disables encryption by doing that.

One of the issues recommended to implement these:

'encryption.legacy_format_support' => false,
'encryption.key_storage_migrated' => false,

But I had no luck with those.

Steps to reproduce

Install Nextcloud (I've seen the issue on both the archive method setup and nextcloud-snap)
Attach to an S3-compatible provider (I am using Wasabi)
Enable Default Encryption Module in Apps
Enable Server Side Encryption and Encrypt the Home Storage (should be default I believe)
Upload a small text file
Confirm the text file is encrypted on Wasabi
Upload a 10 MB file
Confirm the file is not encrypted on Wasabi

Expected behavior

Expected behavior is for files to automatically encrypt when uploaded.

Installation method

Community Manual installation with Archive

Nextcloud Server version

27

Operating system

Debian/Ubuntu

PHP engine version

PHP 8.2

Web server

Apache (supported)

Database engine version

MariaDB

Is this bug present after an update or on a fresh install?

Fresh Nextcloud Server install

Are you using the Nextcloud Server Encryption module?

Encryption is Enabled

What user-backends are you using?

Default user-backend (database)
LDAP/ Active Directory
SSO - SAML
Other

Configuration report

{
    "system": {
        "trashbin_retention_obligation": "30, 90",
        "remember_login_cookie_lifetime": 1296000,
        "filelocking.enabled": true,
        "trusted_domains": [
            "localhost",
            "***REDACTED***.com",
        ],
        "lost_password_link": "disabled",
        "overwriteprotocol": "https",
        "htaccess.RewriteBase": "\/",
        "versions_retention_obligation": "7, 30",
        "trusted_proxies": "***REMOVED SENSITIVE VALUE***",
        "twofactor_enforced": "true",
        "twofactor_enforced_groups": [
            "admin"
        ],
        "twofactor_enforced_excluded_groups": [],
        "mail_smtpmode": "smtp",
        "mail_smtpsecure": "ssl",
        "mail_sendmailmode": "smtp",
        "mail_smtpauthtype": "LOGIN",
        "mail_from_address": "***REMOVED SENSITIVE VALUE***",
        "mail_domain": "***REMOVED SENSITIVE VALUE***",
        "mail_smtphost": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpport": "465",
        "mail_smtpauth": 1,
        "mail_smtpname": "***REMOVED SENSITIVE VALUE***",
        "mail_smtppassword": "***REMOVED SENSITIVE VALUE***",
        "app_install_overwrite": [
            "limit_login_to_ip"
        ],
        "objectstore": {
            "class": "\\OC\\Files\\ObjectStore\\S3",
            "arguments": {
                "bucket": "REDACTED.storage",
                "autocreate": true,
                "key": "***REMOVED SENSITIVE VALUE***",
                "secret": "***REMOVED SENSITIVE VALUE***",
                "hostname": "s3.us-central-1.wasabisys.com",
                "port": 443,
                "use_ssl": true,
                "region": "us-central-1",
                "use_path_style": false,
                "verify_bucket_exists": false
            }
        },
        "memcache.local": "\\OC\\Memcache\\APCu",
        "memcache.distributed": "\\OC\\Memcache\\Redis",
        "redis": {
            "host": "***REMOVED SENSITIVE VALUE***",
            "port": 0
        },
        "maintenance": false,
        "theme": "",
        "simpleSignUpLink.shown": false,
        "default_phone_region": "us",
        "default_locale": "en_US",
        "default_timezone": "America\/Chicago",
        "defaultapp": "files",
        "loglevel": 2,
        "overwrite.cli.url": "https:\/\/REDACTED.com",
        "data-fingerprint": "***REDACTED***",
        "updater.secret": "***REMOVED SENSITIVE VALUE***",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "dbtype": "mysql",
        "version": "27.1.4.1",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbport": "",
        "dbtableprefix": "oc_",
        "mysql.utf8mb4": true,
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "installed": true,
        "instanceid": "***REMOVED SENSITIVE VALUE***"
    }
}

List of activated Apps

Enabled:
  - activity: 2.19.0
  - audioplayer: 3.4.0
  - bruteforcesettings: 2.7.0
  - circles: 27.0.1
  - cloud_federation_api: 1.10.0
  - comments: 1.17.0
  - contactsinteraction: 1.8.0
  - dashboard: 7.7.0
  - dav: 1.27.0
  - encryption: 2.15.0
  - federatedfilesharing: 1.17.0
  - federation: 1.17.0
  - files: 1.22.0
  - files_external: 1.19.0
  - files_pdfviewer: 2.8.0
  - files_reminders: 1.0.0
  - files_rightclick: 1.6.0
  - files_sharing: 1.19.0
  - files_trashbin: 1.17.0
  - files_versions: 1.20.0
  - firstrunwizard: 2.16.0
  - limit_login_to_ip: 4.0.0
  - logreader: 2.12.0
  - lookup_server_connector: 1.15.0
  - nextcloud_announcements: 1.16.0
  - notifications: 2.15.0
  - oauth2: 1.15.1
  - password_policy: 1.17.0
  - photos: 2.3.0
  - privacy: 1.11.0
  - provisioning_api: 1.17.0
  - recommendations: 1.6.0
  - related_resources: 1.2.0
  - serverinfo: 1.17.0
  - settings: 1.9.0
  - sharebymail: 1.17.0
  - support: 1.10.0
  - suspicious_login: 5.0.0
  - systemtags: 1.17.0
  - text: 3.8.0
  - theming: 2.2.0
  - theming_customcss: 1.15.0
  - twofactor_backupcodes: 1.16.0
  - twofactor_totp: 9.0.0
  - unsplash: 2.2.1
  - updatenotification: 1.17.0
  - user_status: 1.7.0
  - viewer: 2.1.0
  - weather_status: 1.7.0
  - workflowengine: 2.9.0
Disabled:
  - admin_audit: 1.17.0
  - geoblocker: 0.5.11 (installed 0.5.11)
  - survey_client: 1.15.0 (installed 1.15.0)
  - user_ldap: 1.17.0

Nextcloud Signing status

No errors have been found.

Nextcloud Logs

Will post below

Additional info

No response

The text was updated successfully, but these errors were encountered:

asheroto · 2023-12-03T16:39:19Z

Other software info

php -m

[PHP Modules]
apcu
bcmath
bz2
calendar
Core
ctype
curl
date
dom
exif
FFI
fileinfo
filter
ftp
gd
gettext
gmp
hash
iconv
igbinary
imagick
intl
json
libxml
mbstring
mysqli
mysqlnd
openssl
pcntl
pcre
PDO
pdo_mysql
Phar
posix
random
readline
redis
Reflection
session
shmop
SimpleXML
sockets
sodium
SPL
standard
sysvmsg
sysvsem
sysvshm
tokenizer
xml
xmlreader
xmlwriter
xsl
Zend OPcache
zip
zlib

[Zend Modules]
Zend OPcache

php configuration

mbstring.func_overload=0
always_populate_raw_post_data=-1
default_charset='UTF-8'
output_buffering=0
memory_limit=640M
upload_max_filesize=16G
post_max_size=16G
max_input_time=3600
max_execution_time=3600
opcache.save_comments=1
opcache.revalidate_freq=60
opcache.validate_timestamps=0
opcache.jit=1255
opcache.jit_buffer_size=128M
opcache.memory_consumption=256
opcache.interned_strings_buffer=128

MySQL is configured per the recommended settings.

Log info

The first set of log messages only appear when running occ encryption:encrypt-all. I do not see any errors at all when uploading.

The second set of log messages only appear when running occ encryption:scan:legacy-format. I have implemented the OpenSSL workaround as mentioned in #32003 but that did not fix the issue for me even after restarting the php8.2-fpm and apache2 - the same error message appears.

Security & setup warnings

None

Summary

Honestly I'm not doing anything fancy. 😊 You can quickly replicate the problem for yourself if you simply perform the steps to reproduce. You can sign up with Wasabi for free in 5 minutes. It's usage-based billing. I installed a clean install of Nextcloud, php 8.2, Redis, MySQL, APCu, Apache. No service errors nor log errors from any of those where applicable.

As a side note, I was never able to get E2EE + SSE + S3 working. Only SSE + S3. But now that has seemed to stop as well unless manually engaged. :-/

I am hopeful for a resolution. I appreciate your time on this, thank you! 😁

Logs

logs.txt

Also

I also donate my time to many open source projects so I understand that time is valuable. So I really appreciate your help on this! I might be able to submit a PR but am unfamiliar with the inner-workings of Nextcloud from a developer standpoint. 😊

solracsf · 2023-12-03T17:35:38Z

I personally don't recommend using Nextcloud Encyption module with S3.
Instead, use SSE-C with custom provided keys. https://docs.nextcloud.com/server/latest/admin_manual/configuration_files/primary_storage.html#s3-sse-c-encryption-support

asheroto · 2023-12-04T20:39:44Z

Thanks, I will try that as a workaround.

The disadvantage with that, of course, is that the files are transmitted unencrypted (encrypted in the connection, but unencrypted in memory). Although the S3 provider only stores the key in memory, it could be possible for bad actors within the S3 provider itself to intercept files or the key. This is of course highly unlikely but in theory possible, so is not the preferred method. This should work for now though until Nextcloud resolves the encryption issues with S3.

It seems that it is encrypting data but only when initiated manually through the occ encryption:encrypt-all command, so I think the bugs are pretty close to being resolved, I hope. 😊

Going to leave the ticket open since the issue itself is not resolved yet.

asheroto · 2023-12-04T21:10:40Z

I personally don't recommend using Nextcloud Encyption module with S3. Instead, use SSE-C with custom provided keys. https://docs.nextcloud.com/server/latest/admin_manual/configuration_files/primary_storage.html#s3-sse-c-encryption-support

Just tried this. Sadly another Nextcloud bug. Can we just use this issue to track the encryption issues? Seems to be all related to each other.

Could not decrypt key

Log errors for that:
https://gist.github.com/asheroto/b4f13d528b48a304a0702d939c958494

Steps to reproduce:
1.) Use sse_c_key with Wasabi

Should be the same as AWS:
https://knowledgebase.wasabi.com/hc/en-us/articles/4414850567963-How-does-SSE-C-Encryption-work-with-Wasabi-

This is sad that it doesn't work on a new installation. :-(

Let me know if I should create another ticket. All of these issues seem to be related.

asheroto · 2023-12-04T21:18:46Z

Oops, so when using SSE after having used encryption on Nextcloud, you have to keep the Default Encryption Module, but disable encryption. Otherwise you'll get that error.

Ideally the documentation should mention this, and instead of just saying Could not decrypt key it could help you and say Could not decrypt key. New SSE-C seen, try disabling encryption. or something like that. 😊

Thanks, will try this for now but hoping for a resolution on the main encryption method. 😊

asheroto · 2023-12-05T00:00:51Z

Possibly related to #41704. In reinstalling Nextcloud today (again), I found that error on a new installation.

asheroto added 0. Needs triage Pending check for reproducibility or if it fits our roadmap bug labels Dec 3, 2023

szaimen added the 27-feedback label Dec 3, 2023

joshtrichards added feature: encryption (server-side) feature: object storage labels Dec 3, 2023

asheroto mentioned this issue Feb 19, 2024

Invalid private key for encryption app. Please update your private key password in your personal settings to recover access to your encrypted files #8546

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Bug]: 🪲 Server side encryption does not encrypt files with S3 primary storage, related error messages on encrypt-all and scan:legacy-format 😢 #41992

[Bug]: 🪲 Server side encryption does not encrypt files with S3 primary storage, related error messages on encrypt-all and scan:legacy-format 😢 #41992

asheroto commented Dec 3, 2023 •

edited

Loading

asheroto commented Dec 3, 2023 •

edited

Loading

solracsf commented Dec 3, 2023 •

edited

Loading

asheroto commented Dec 4, 2023 •

edited

Loading

asheroto commented Dec 4, 2023

asheroto commented Dec 4, 2023 •

edited

Loading

asheroto commented Dec 5, 2023

[Bug]: 🪲 Server side encryption does not encrypt files with S3 primary storage, related error messages on encrypt-all and scan:legacy-format 😢 #41992

[Bug]: 🪲 Server side encryption does not encrypt files with S3 primary storage, related error messages on encrypt-all and scan:legacy-format 😢 #41992

Comments

asheroto commented Dec 3, 2023 • edited Loading

⚠️ This issue respects the following points: ⚠️

Bug description

Preface

Background

Troubleshooting

Similar issues

Steps to reproduce

Expected behavior

Installation method

Nextcloud Server version

Operating system

PHP engine version

Web server

Database engine version

Is this bug present after an update or on a fresh install?

Are you using the Nextcloud Server Encryption module?

What user-backends are you using?

Configuration report

List of activated Apps

Nextcloud Signing status

Nextcloud Logs

Additional info

asheroto commented Dec 3, 2023 • edited Loading

Other software info

Log info

Security & setup warnings

Summary

Logs

Also

solracsf commented Dec 3, 2023 • edited Loading

asheroto commented Dec 4, 2023 • edited Loading

asheroto commented Dec 4, 2023

asheroto commented Dec 4, 2023 • edited Loading

asheroto commented Dec 5, 2023

asheroto commented Dec 3, 2023 •

edited

Loading

asheroto commented Dec 3, 2023 •

edited

Loading

solracsf commented Dec 3, 2023 •

edited

Loading

asheroto commented Dec 4, 2023 •

edited

Loading

asheroto commented Dec 4, 2023 •

edited

Loading