Partial downloads from S3 Primary Storage

[2xaronl]

Steps to reproduce

1. Download a file larger than ~50MB through web interface

Expected behaviour

File should download

Actual behaviour

Download starts but transfer rate trickles down to ~4KB/s and eventually stops with failure

Server configuration

Operating system: Ubuntu 18.04

Web server: Apache 2.4.29

Database: MySQL 5.6.39

PHP version: 7.2

Nextcloud version: 14.0.3

Updated from an older Nextcloud/ownCloud or fresh install: Updated from 13

Where did you install Nextcloud from: Direct download

Signing status:

Signing status

No errors have been found.

List of activated apps:

App list

The current PHP memory limit is below the recommended value of 512MB.
Enabled:
  - accessibility: 1.0.1
  - activity: 2.7.0
  - bruteforcesettings: 1.1.0
  - cloud_federation_api: 0.0.1
  - comments: 1.4.0
  - dav: 1.6.0
  - federatedfilesharing: 1.4.0
  - federation: 1.4.0
  - files: 1.9.0
  - files_pdfviewer: 1.3.2
  - files_rightclick: 0.8.4
  - files_sharing: 1.6.2
  - files_texteditor: 2.6.0
  - files_trashbin: 1.4.1
  - files_versions: 1.7.1
  - files_videoplayer: 1.3.0
  - firstrunwizard: 2.3.0
  - gallery: 18.1.0
  - logreader: 2.0.0
  - lookup_server_connector: 1.2.0
  - nextcloud_announcements: 1.3.0
  - notifications: 2.2.1
  - oauth2: 1.2.1
  - password_policy: 1.4.0
  - provisioning_api: 1.4.0
  - serverinfo: 1.4.0
  - sharebymail: 1.4.0
  - support: 1.0.0
  - survey_client: 1.2.0
  - systemtags: 1.4.0
  - theming: 1.5.0
  - twofactor_backupcodes: 1.3.1
  - updatenotification: 1.4.1
  - workflowengine: 1.4.0
Disabled:
  - admin_audit
  - encryption
  - external
  - files_accesscontrol
  - files_external
  - user_external
  - user_ldap

Nextcloud configuration:

Config report

The current PHP memory limit is below the recommended value of 512MB.
{
    "system": {
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "objectstore": {
            "class": "OC\\Files\\ObjectStore\\S3",
            "arguments": {
                "bucket": "***REMOVED SENSITIVE VALUE***",
                "autocreate": true,
                "key": "***REMOVED SENSITIVE VALUE***",
                "secret": "***REMOVED SENSITIVE VALUE***",
                "use_ssl": true,
                "region": "us-west-2",
                "use_path_style": true
            }
        },
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "nextcloud.***REMOVED SENSITIVE VALUE***.com"
        ],
        "debug": true,
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "overwrite.cli.url": "https:\/\/nextcloud.***REMOVED SENSITIVE VALUE***.com\/",
        "htaccess.RewriteBase": "\/",
        "dbtype": "mysql",
        "version": "14.0.3.0",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbport": "",
        "dbtableprefix": "oc_",
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "installed": true,
        "logtimezone": "America\/Los_Angeles",
        "log_type": "owncloud",
        "logfile": "\/var\/log\/nextcloud.log",
        "loglevel": "0",
        "cron_log": true,
        "log_rotate_size": "104857600",
        "memcache.local": "\\OC\\Memcache\\APCu",
        "mail_smtpmode": "smtp",
        "mail_smtpauthtype": "LOGIN",
        "mail_smtpsecure": "ssl",
        "mail_smtphost": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpport": "465",
        "mail_smtpauth": 1,
        "mail_from_address": "***REMOVED SENSITIVE VALUE***",
        "mail_domain": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpname": "***REMOVED SENSITIVE VALUE***",
        "mail_smtppassword": "***REMOVED SENSITIVE VALUE***",
        "maintenance": false,
        "auth.bruteforce.protection.enabled": false,
        "updater.release.channel": "stable"
    }
}

Are you using external storage, if yes which one: S3 Primary

Are you using encryption: no

Are you using an external user-backend, if yes which one: No

Client configuration

Browser: Firefox 61 / Chrome 69

Operating system: Windows 10 x64

Logs

The logs don't appear to have any pertinent info

More info posted here: https://help.nextcloud.com/t/downloads-from-nextcloud-stalling-s3-primary-storage/39191

[nextcloud-bot]

GitMate.io thinks possibly related issues are #4730 (Storage), #5928 (Support for multiple primary storage?), #3748 (Primary S3 Storage Backend + Encryption problem with file size, mimetype and first download failed), #11637 (No upload to shared folder possible after update to 14.0.1 (S3 primary storage)), and #10022 (S3 Primary storage backup).

[fetra1]

I have pretty much the same problem. My downloads stay at 0 - 3,4 kbit/s after downloading 30 - 60 MB of a big file.
Wireshark says that every 2-3 secs one package is downloading.
I have tested to download a file directly from apache which worked just fine. Downloading files via php code is working as well. Same for downloads directly from S3.
Uploads work fine.
A workaround or fix would be nice, because this is preventing us from using the server!

Edit: We mounted local directories as external storage and put our files there. Downloads are working now. As soon as we try to download anything from the S3 Bucket the problem is back. It has definitely something to do with the S3 Implementation of Nextcloud. Moving the files from S3 to the local directories worked without problems. So Nextcloud is able to get the files but has problems bringing them to the client.
Help is still appreciated because storing the files on EBS volumes is much more expensive than using S3!

[2xaronl]

@fetra1 Have you made any progress on finding a solution to the S3 issues?

[fetra1]

I completely migrated to EBS storage. This solved the problem. After updating to 14.0.4 I mounted S3 as external volume and retried it which worked without any problems.
I'm not sure either updating to 14.0.4 or migrating to EBS solved the problem. But we dont migrate back to S3 as S3 with nextcloud seems to be really buggy.

[2xaronl]

This is still an issue with Nextcloud 15.0.2. Large uploads and downloads to S3 primary storage are impacted.

[2xaronl]

After switching from Apache to nginx, increasing the RAM on my server from 512MB to 2GB, increasing proxy buffer and memory limits and timeouts substantially, creating a dedicated partition for temporary files for buffering, updating to Nextcloud 16.0.0 and applying the fix posted here: #13829 it appears as though uploads and downloads of large files is working as expected. I'm not sure what exactly fixed the issue or if it was a combination of different things that I did but I'm glad it's working and I hope it stays working.

Large (~3GB) multipart uploads are unreliable, failing about 50% of the time.

[skjnldsv]

Still happening?

[ghost]

This issue has been automatically marked as stale because it has not had recent activity and seems to be missing some essential information. It will be closed if no further activity occurs. Thank you for your contributions.