Decryption corrupted all files — serious file loss

[mmaedler]

Steps to reproduce

1. Enable server-side encryption via occ encryption encrypt:all
2. Disable server-side encryption via occ encryption decrypt:all
3. All files being opened trigger "Bad Signature error in log" and seem corrupted

Expected behaviour

Files should be decrypted and accessible

Actual behaviour

Files are corrupted and cannot be opened anymore. Due to that I have lost important files.

Server configuration

Operating system: Ubuntu 16.04 server

Web server: nginx

Database: mysql

PHP version: 7.0

Nextcloud version: 12.0.4

Updated from an older Nextcloud/ownCloud or fresh install: Updated

Where did you install Nextcloud from:

Signing status:

Signing status

No errors have been found.

List of activated apps:

App list

Enabled:
  - activity: 2.5.2
  - admin_audit: 1.2.0
  - admin_notifications: 1.0.1
  - announcementcenter: 3.1.1
  - bruteforcesettings: 1.0.3
  - calendar: 1.5.7
  - comments: 1.2.0
  - contacts: 2.0.1
  - dav: 1.3.0
  - encryption: 1.6.0
  - external: 2.0.3
  - federatedfilesharing: 1.2.0
  - federation: 1.2.0
  - files: 1.7.2
  - files_accesscontrol: 1.2.5
  - files_automatedtagging: 1.2.2
  - files_external: 1.3.0
  - files_pdfviewer: 1.1.1
  - files_sharing: 1.4.0
  - files_texteditor: 2.4.1
  - files_trashbin: 1.2.0
  - files_versions: 1.5.0
  - files_videoplayer: 1.1.0
  - firstrunwizard: 2.1
  - gallery: 17.0.0
  - logreader: 2.0.0
  - lookup_server_connector: 1.0.0
  - nextcloud_announcements: 1.1
  - notifications: 2.0.0
  - oauth2: 1.0.5
  - password_policy: 1.2.2
  - provisioning_api: 1.2.0
  - quota_warning: 1.1.1
  - serverinfo: 1.2.0
  - sharebymail: 1.2.0
  - socialsharing_email: 1.0.3
  - survey_client: 1.0.0
  - systemtags: 1.2.0
  - theming: 1.3.0
  - twofactor_backupcodes: 1.1.1
  - updatenotification: 1.2.0
  - workflowengine: 1.2.0

Nextcloud configuration:

Config report

{
    "system": {
        "instanceid": "occ76c8edd49",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "datadirectory": "\/var\/www\/nextcloud\/data",
        "dbtype": "mysql",
        "version": "12.0.4.3",
        "dbname": "owncloud",
        "dbhost": "127.0.0.1",
        "dbtableprefix": "oc_",
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "installed": true,
        "loglevel": 3,
        "logtimezone": "Europe\/Berlin",
        "maintenance": false,
        "theme": "",
        "appstoreenabled": true,
        "appstoreurl": "https:\/\/apps.nextcloud.com\/api\/v0",
        "trusted_domains": [
            "oc.betaserv.net"
        ],
        "mail_smtpmode": "php",
        "mail_smtpsecure": "ssl",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "forcessl": true,
        "memcache.local": "\\OC\\Memcache\\APCu",
        "memcache.locking": "\\OC\\Memcache\\Redis",
        "redis": {
            "host": "\/run\/redis\/redis.sock",
            "port": 0,
            "dbindex": 0,
            "timeout": 1.5
        },
        "appstore.experimental.enabled": true,
        "trashbin_retention_obligation": "auto",
        "updater.release.channel": "stable",
        "mail_from_address": "Nextcloud",
        "mail_domain": "",
        "mail_smtpauthtype": "LOGIN",
        "mail_smtpauth": 1,
        "mail_smtphost": "",
        "mail_smtpport": "587",
        "preview-libreoffice-path": "\/lib\/libreoffice\/program\/soffice",
        "singleuser": true,
        "updatechecker": true,
        "updater.server.url": "https:\/\/updates.nextcloud.com\/updater_server\/",
        "token_auth_enforced": true,
        "overwrite.cli.url": "https:\/\/oc.betaserv.net"
    }
}

**Are you using encryption:** yes and no

[AlexCloudDev]

hi, same here :(

we ran our nc instance with enabled server-side encryption since owncloud 7 times and disabled it 3 days ago via
"occ encryption decrypt:all"

Currently we are on nc13, php7.
User data encryption is still enabled.

After that we get "OCP\Encryption\Exceptions\GenericEncryptionException: Bad Signature" errors at every file we try to open. Files are still there but inaccessible.
The "occ encryption decrypt:all" command finished without errors.

pls help
thx

log:

OCP\Encryption\Exceptions\GenericEncryptionException: Bad Signature
/var/www/html/apps/encryption/lib/Crypto/Crypt.php - line 465: OCA\Encryption\Crypto\Crypt->checkSignature('IbvrdqBYkdwoRjr...', '+dBH\x0E\xCA\xBD\xD4U\xC2\xAD>\x0E\xA7z...', 'c5578ba7708b5b6...')
/var/www/html/apps/encryption/lib/Crypto/Encryption.php - line 380: OCA\Encryption\Crypto\Crypt->symmetricDecryptFileContent('IbvrdqBYkdwoRjr...', '+dBH\x0E\xCA\xBD\xD4U\xC2\xAD>\x0E\xA7z...', 'AES-256-CTR', 0, 0)
/var/www/html/lib/private/Files/Stream/Encryption.php - line 464: OCA\Encryption\Crypto\Encryption->decrypt(*** sensitive parameters replaced )
/var/www/html/lib/private/Files/Stream/Encryption.php - line 295: OC\Files\Stream\Encryption->readCache()
[internal function] OC\Files\Stream\Encryption->stream_read(8192)
/var/www/html/3rdparty/icewind/streams/src/Wrapper.php - line 83: fread(Resource id #42, 8192)
/var/www/html/3rdparty/icewind/streams/src/CallbackWrapper.php - line 91: Icewind\Streams\Wrapper->stream_read(8192)
[internal function] Icewind\Streams\CallbackWrapper->stream_read(8192)
/var/www/html/lib/private/Files/View.php - line 425: fread(Resource id #45, 8192)
/var/www/html/lib/private/legacy/files.php - line 310: OC\Files\View->readfile('//ts3.txt')
/var/www/html/lib/private/legacy/files.php - line 122: OC_Files getSingleFile(Object(OC\Files\View), '/', 'ts3.txt', Array)
/var/www/html/apps/files/ajax/download.php - line 64: OC_Files get('/', 'ts3.txt', Array)
/var/www/html/lib/private/Route/Route.php - line 155: require_once('/var/www/html/a...')
[internal function] OC\Route\Route->OC\Route{closure}( sensitive parameters replaced ***)
/var/www/html/lib/private/Route/Router.php - line 297: call_user_func(Object(Closure), Array)
/var/www/html/lib/base.php - line 998: OC\Route\Router->match('/apps/files/aja...')
/var/www/html/index.php - line 37: OC handleRequest()
{main}

[mmaedler]

Hi —
is there any update regarding this topic? I really need my files back...
Thanks!

[tessus]

Has anyone found a solution? Is this being investigated?

Now and then I truly believe that priorities are totally screwed up with this project. Sometimes a proper icon placement seems more important than making a basic feature work.

[blizzz]

@schiessle

[FlorianFranzen]

I am facing the same issue on nextcloud 13.0.1.

The problem seems to arises when encryption:decrypt-all is run in maintenance mode, during which the encryption module is actually disabled.

As a result, you will not be asked for the recovery password before the decryption and none of the files will be decrypted, but marked as such in the database, resulting in aforementioned error.

To reproduce the issue I just tested this with a fresh docker-based installation:

1. Enable encryption.
2. Upload a new file
3. Go to maintenance mode
4. Run decrypt-all

Result: File no longer accessible with error above (after leaving maintenance mode).

To get access to the file again after the failed decryption it was enough to set the files encrypted column in the filecache table back to 1 (i.e. update oc_filecache set encrypted = 1 where fileid = <FILE_ID> if you have the file id).

@schiessle Is there a quick way to detect these files and to fix their DB entries without having to restore backups?

[mmaedler]

Interesting find @FlorianFranzen ! Is it possible to simple change the encrypted-column back to 1 for all files (since I guess none of them is accessible at the moment) in one go?

Thanks!

[FlorianFranzen]

Disclaimer: I am not an nextcloud developer and you should have backups of all your data and database before you start messing with nextcloud's internal structure.

@mmaedler Yes, you could. The problem is just, that not all entries in the file cache are files (but dirs, etc.), whose encrypted flag should probably not be set.

I am working at a fix at the moment that avoids working on the database directly. I failed to get any help from any actual developers, but given the projects bad reputation that is not surprising at all.

It seems that even if the file is not marked encrypted in file cache, the encryption stream wrapper is called somehow (probably based on file content) but then fails because the file cache marks the file as not encrypted.

[AlexCloudDev]

Is there a way to decrypt the files via "occ encryption decrypt:all" without entering the maintenance mode to avoid the problem?

Thx for your help guys :)

[FlorianFranzen]

@AlexCloudDev Yes, some quick test seems to suggest that it is save. decrypt-all actually enables maintenance mode during the decryption, so there is no need for it anyway.

[mmaedler]

I tried what you suggested earlier and when running occ decrypt-all I get

Server side encryption not enabled. Nothing to do.

I guess this is because I ran the decrypt-all command before (and then ended up with this mess of files). Shall I enable encryption again?

Thanks,

Moritz

[FlorianFranzen]

@mmaedler: Just enable encryption for a brief moment, decrypt-all will disable it again anyway. Enabling encryption does only mean, that all new files added will be encrypted.

[FlorianFranzen]

@schiessle: This is quite a serious bug, that can be easily replicate in a few steps in a fresh install. Care to take a look or to at least comment on this issue?

[FlorianFranzen]

@MorrisJobke @rullzer You seem to be some sort of nextcloud maintainer. I already tried to talk to people on IRC but nobody seems to care. Would anybody care to comment on this issue?

[MorrisJobke]

cc @nextcloud/encryption

[AnianZ]

Bump to remove stale label I guess. This is still relevant and can cause serious headaches. I don't get why nothing is done about it. If there is not immediate fix at least the occ encryption decrypt:all should be deactivated while the problem persists.

[schiessle]

@AnianZ can you test it with the Nextcloud 14 beta 3? We added quite some encryption fixes to it, so chance are high that they will also resolve this issue... Thanks! https://nextcloud.com/blog/nextcloud-14-beta-3-is-here-time-for-testing-and-a-chance-to-win-a-t-shirt/

[riussi]

Thank God for this advice:

update oc_filecache set encrypted = 1

Saved me when the decrypt all did nothing but mark the files decrypted causing all files to show up as corrupt. Changed them to encrypted = 1 in the db and managed to recover them.

This is a huge problem and can cause serious loss of data.

[FlorianFranzen]

@schiessle It is an easy to reproduce bug, that has been filed with high importance with you for more than half a year and started being reported as early as at least nine month ago. You are the main encryption developer. How about you test your own code for a change?

[RubenHoms]

@FlorianFranzen I can't thank you enough for figuring this out. Your advice has literally saved 5 years worth of data. ❤️

[akalypse]

@AnianZ can you test it with the Nextcloud 14 beta 3? We added quite some encryption fixes to it, so chance are high that they will also resolve this issue... Thanks! https://nextcloud.com/blog/nextcloud-14-beta-3-is-here-time-for-testing-and-a-chance-to-win-a-t-shirt/

@schiessle : The new version changed nothing in this regards. I am still not able to decrypt my files using occ occ encryption decrypt:all

[akalypse]

Alright, after studying a bit the code, I found out that:

Precondition:

• You don't have a master key / recovery key specified

Then:

• you SHALL NOT BE in Maintenance mode
• it is obligatory to specify the User when running the decrypt:all command
• you specify the user's password when asked

Then the decryption happens and you can read your files in clear text again.

This should be documented in the latest documentation, in my opinion.

[RubenHoms]

@iegtcamnp I'm also running the snap and managed to decrypt my files with a workaround. You've almost got it down, you just need to skip step 2, maintenance mode needs to be disabled when you're decrypting your files. This is because while decrypting the files it checks if the encryption module is active and when the Nextcloud instance is in maintenance mode it returns false and just assumes that decryption was successful.
Good luck, hope you can work it out.

[RubenHoms]

Fix has just been merged into master. 👍

[ghost]

just to let you know. this is, in my opinion, a major bug and should be fixed as soon as possible. I am on 16.0.1.1 and have still the problem

[FlorianFranzen]

@ybaumy 16.0.1 was released before the fix was merged, so it should be included in 16.0.2.

[kesselb]

@ybaumy 16.0.1 was released before the fix was merged, so it should be included in 16.0.2.

Nextcloud 17.

[ghost]

@kesselb If this comes out in nextcloud 17 latest then do other users a favor and include some information in the documenation that decrypt-all is broken. Or point to a workaround.

[kesselb]

@ybaumy Good point 👍 Pull Requests are always welcome: https://github.com/nextcloud/documentation

[ghost]

@kesselb well I won't add it to the documentation since I won't use nextcloud much longer. the decrypt-all bug is just one of the annoyances. but I am pretty sure somebody will or not, since you pretty much do not seem to care.

[kesselb]

since you pretty much do not seem to care.

I see you're frustrated, but don't take it out on me. I'm a user like you and do contributions in my spare time 😞

[ghost]

@kesselb Hey man never mind. I am beyond frustration. Dealing with badly documented OSS software and bugs for over 20 years. You spend hours or sometimes days identifying a problem and you have to wait and wait to get a fix for it. It is a completely normal behavior. Even if users lose their data or have to restore 50TB now, like in my case.
Sometimes you also realize that even if you have purchased an enterprise product documentation is just at the same stage as for normal users.

[Ciangi]

Hey,

are you sure that the problem has been resolved?

After latest upgrade of nextcloud i still have some files not decrypted...
I used occ enryption:decrypt-all on 16.0.1 Nextcloud version with maintenance mode enabled.

I noticed that the files, that are not decrypted, their 'path' in database is still with 'files_encyption/%'
when i tried to update some 'fieldid' with enrypted=1 and then try to do occ enryption:decrypt-all,
on the database record got encrypted to 0 but in nextcloud it is still encrypted...

Thanks for any help.

[RubenHoms]

Not sure what situation you're in @Ciangi , but the fix I made for this only makes it impossible to decrypt while you're in maintenance mode. Decrypting when maintenance mode is enabled made the encryption modules be unavailable which caused the corruption in this case.

If you already had files in this situation, then upgraded and tried to decrypt again it will not do anything for that. If you need to fix that situation take a look at this comment and my reply below that to fix that issue.

[yahesh]

@ybaumy @Ciangi @iegtcamnp @mmaedler @tessus I don't know if this is still relevant for you but we've written a tool that allows you to decrypt individual files if you still have your Nextcloud data directory and configuration file. It supports master key encrypted files, user key encrypted files (you additionally need the user passwords) and recovery key encrypted files (you additionally need the recovery password): decrypt-file.php

[kwiatekk]

Hi there,
I'm having recently the same problem with encrypted files with nextcloud(snap) installation with Ubuntu 18.04.
Could anyone help me to overcome this problem?
Where to start?
Thanks in advance.
Kristof

[JB1985]

I am also here because I also have the problem that not all files are decrypted, too. I also do not know how to save the data. Update to 17.0.1 and to make decrypt again?

I'm stinking why it was not documented that the decryption ist buggy.

[kwiatekk]

update oc_filecache set encrypted = 1 where fileid = <FILE_ID>;
JB1985 have you tried to type this command?
Or maybe you know what to do with it?

[JB1985]

kwiatekk tell me how to find the fileid of the files that there are not yet decryptet?!

[kwiatekk]

Do not, yet I'm still fighting with that.

…

On Sat, 23 Nov 2019, 22:08 JB1985, ***@***.***> wrote: kwiatekk tell me how to find the fileid of the files that there are not yet decryptet?! — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#8311?email_source=notifications&email_token=AJDCURBDV44QBQOBX66JS7DQVGLU3A5CNFSM4EQIGEX2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEE75WHI#issuecomment-557832989>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AJDCURDMPEWZLFOJKQE3HC3QVGLU3ANCNFSM4EQIGEXQ> .

[JB1985]

There are some files that can not be decrypted. I have try with @yahesh decrypt-file.php but still not work.

[yahesh]

@kwiatekk @JB1985 I wouldn't advise to directly modify the database, but rather restore a backup of your server from before you tried to decrypt all files and just download the files from Nextcloud after the restore.

The encrypted database field actually isn't a boolean but an integer that also denotes the file version of an encrypted file which is relevant to calculate the MACs/"signatures" of the encrypted files. If you have to fiddle around with the database in order to rescue your encrypted files then it would be advisable to also set the encryption_skip_signature_check configuration value of your Nextcloud instance to true.

[phonon112358]

I encountered the same issue in NC 18.0.1 when following the documentation!!!

[ffs69]

Hi,

I am facing the same problem here, many of my .jpg and .txt files have become corrupted. And unfortunately I don't have a recent backup.

Is there a working solution since? I'm surprised the problem is still happening.

@yahesh is it possible and how can I use rescue/decrypt-all-files.php on a shared web hosting?

Many thanks in advance, I am in depression!

C.