Sending Large Files

[solepixel]

This issue is related to uploading larger files, such as 800MB and 1GB. The code I have for sending ALL files is the same:

public static function upload( $file, $folder_id ) {
	self::increase_all_the_limits();

	try {
		$folder = self::$client->getDriveItemById( $folder_id );
	} catch ( \Exception $e ) {
		self::log( 'There was an error getting OneDrive file properties for `' . $folder_id . '`: ' . $e->getMessage(), 'error' );
		return false;
	}

	try {
		$upload = $folder->startUpload( basename( $file ), fopen( $file, 'r' ), $args );
	} catch ( \Exception $e ) {
		self::log( 'Error: Could not initiate upload for `' . basename( $file ) . '`. Details: ' . $e->getMessage(), 'error' );
		return false;
	}

	try {
		$new_file = $upload->complete();
	} catch ( \Exception $e ) {
		self::log( 'Error: OneDrive upload failed for `' . basename( $file ) . '`. Details: ' . $e->getMessage(), 'error' );
		return false;
	}

	// Don't use isset or empty here.
	if ( ! $new_file->id ) {
		self::log( 'OneDrive upload status for `' . basename( $file ) . '` was missing ID property: ' . print_r( $new_file->id, true ), 'error' );
		return false;
	}

	return true;
}

During the $upload->complete() step, the entire file contents are retrieved, no matter what the file size. See:

      while (!$stream->eof()) {
            $rangeStream = new LimitStream($stream, $rangeSize, $offset);
            $rangeSize   = $rangeStream->getSize();
            $body        = $rangeStream->getContents();
            $rangeFirst  = $offset;
            $offset += $rangeSize;
            $rangeLast = $offset - 1;

            $headers = [
                'Content-Length' => $rangeSize,
                'Content-Range'  => "bytes $rangeFirst-$rangeLast/$size",
            ];

            $response = $this
                ->graph
                ->createRequest('PUT', $this->uploadUrl)
                ->addHeaders($headers)
                ->attachBody($body)
                ->execute();

            $status = $response->getStatus();

            if ($status == 200 || $status == 201) {
                $driveItem = $response->getResponseAsObject(DriveItem::class);

                return new DriveItemProxy(
                    $this->graph,
                    $driveItem,
                    $this->driveItemResourceDefinition
                );
            }

            if ($status != 202) {
                throw new \Exception("Unexpected status code produced by 'PUT {$this->uploadUrl}': $status");
            }
        }

Do you have an example of how to utilize the rangeSize option in the startUpload() $args to manually process the upload in chunks? Otherwise, do you have any suggestions? We continue to hit problems with timeouts, memory, and 503 Errors from OneDrive. It would be great to have a "status" returned during $upload->complete() or maybe even something like $upload->getStatus(), so I can close the request, and use the status to continue the upload on a new request. We've done something similar with Google Drive and Dropbox, but using the code above isn't quite cutting it for OneDrive. Thanks!

[krizalys]

@solepixel The code above is supposed to send your file in chunks, eg. this creates a 320KiB chunk:

new LimitStream($stream, $rangeSize, $offset);

Perhaps the code above is suffering from a memory leak on some systems, I will need to stress-test it a bit more.

Can you tell me the OS & PHP version you are using?

[solepixel]

I believe it is sending in chunks, however there is no way to break the chunks into multiple requests, it tries to send the entire file all in the same process, so generally we're hitting 30sec timeout issues.

Tested on MacOS Catalina (10.15.4) and PHP 7.4.2 and PHP 7.3.9 as well as Ubuntu 14.04.6 with PHP 7.0.33 and PHP 7.4.6.

[ahsunali]

@krizalys thanks for the sdk.
I am trying to upload 2GB file and it is taking too long to upload, My hosting allowed max 15 min to a job and it break in the middle. However i can upload the same file to google-drive & ftp in seconds.

Do not know where actually the issue is, either it belongs to OneDrive or upload utility.
Need your help to identify the cause, as if it remains the same we need to close our account with OneDrive
Thanks!