Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added MinIO Storage adapter #51

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
119 changes: 119 additions & 0 deletions src/Storage/Device/MinIO.php
Original file line number Diff line number Diff line change
@@ -0,0 +1,119 @@
<?php

namespace Utopia\Storage\Device;

use Utopia\Storage\Device\S3;

class MinIO extends S3
{
/**
* Regions constants
*
*/
const EU_CENTRAL_1='eu-central-1';
const US_SOUTHEAST_1='us-southeast-1';
const US_EAST_1='us-east-1';
const AP_SOUTH_1='ap-south-1';

/**
* Object Storage Constructor
*
* @param string $root
* @param string $accessKey
* @param string $secretKey
* @param string $protocol
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
* @param string $protocol

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Protocol has to be here, because MinIO can be hosted in an internal Network, where you don't have SSL

* @param string $host
* @param string $bucket
* @param string $region
* @param string $acl
*/
public function __construct(string $root, string $accessKey, string $secretKey, string $protocol, string $host, string $bucket, string $region = self::EU_CENTRAL_1, string $acl = self::ACL_PRIVATE)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
public function __construct(string $root, string $accessKey, string $secretKey, string $protocol, string $host, string $bucket, string $region = self::EU_CENTRAL_1, string $acl = self::ACL_PRIVATE)
public function __construct(string $root, string $accessKey, string $secretKey, string $host, string $bucket, string $region = self::EU_CENTRAL_1, string $acl = self::ACL_PRIVATE)

https://github.com/minio/minio-js/blob/master/src/main/minio.js#L70
Please avoid changing the signature of the constructor. These days it is very uncommon to use HTTP so let's default to HTTPS like the official SDK.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @shimonewman,

thanks for the review. The problem is that in my setup MinIO is in a Self Hosted Private Network so I access it with the internal IP so it don't have an https protocol.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it might be better to adapt all adapters to use and endpoint instead of protocol and hostname for more flexibility. Hardcoding https:// is wrong to begin with.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would leave it for now and make it in another Pull Request

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@groschi24, please create another PR with the proposed changes to the endpoint.

{
parent::__construct($root, $accessKey, $secretKey, $bucket, $region, $acl);

$this->protocol = $protocol;
groschi24 marked this conversation as resolved.
Show resolved Hide resolved
$this->headers['host'] = $host;
}

/**
* Get list of objects in the given path.
*
* @param string $path
*
* @throws \Exception
*
* @return array
*/
public function listObjects($prefix = '', $maxKeys = 1000, $continuationToken = '')
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
public function listObjects($prefix = '', $maxKeys = 1000, $continuationToken = '')
private function listObjects($prefix = '', $maxKeys = 1000, $continuationToken = '')

Please change S3 listObjects() access modifiers to protected in order for you to override it.

{
$uri = '/' . $this->getRoot();
$this->headers['content-type'] = 'text/plain';
$this->headers['content-md5'] = \base64_encode(md5('', true));

$parameters = [
'list-type' => 2,
'prefix' => $prefix,
'max-keys' => $maxKeys,
];
if(!empty($continuationToken)) {
$parameters['continuation-token'] = $continuationToken;
}
$response = parent::call(self::METHOD_GET, $uri, '', $parameters);
return $response->body;
}

/**
* Delete files in given path, path must be a directory. Return true on success and false on failure.
*
* @param string $path
*
* @throws \Exception
*
* @return bool
*/
public function deletePath(string $path): bool
{
$uri = '/' . $this->getRoot();
$continuationToken = '';
do {
$objects = $this->listObjects($path, continuationToken: $continuationToken);
$count = (int) ($objects['KeyCount'] ?? 1);
if($count < 1) {
break;
}
$continuationToken = $objects['NextContinuationToken'] ?? '';
$body = '<Delete xmlns="http://s3.amazonaws.com/doc/2006-03-01/">';

if($count > 1) {
foreach ($objects['Contents'] as $object) {
$body .= "<Object><Key>{$object['Key']}</Key></Object>";
}
} else {
$body .= "<Object><Key>{$objects['Contents']['Key']}</Key></Object>";
}
$body .= '<Quiet>true</Quiet>';
$body .= '</Delete>';
$this->amzHeaders['x-amz-content-sha256'] = \hash('sha256', $body);
$this->headers['content-md5'] = \base64_encode(md5($body, true));
parent::call(self::METHOD_POST, $uri, $body, ['delete'=>'']);
} while(!empty($continuationToken));

return true;
}

/**
* @return string
*/
public function getName(): string
{
return 'MinIO Object Storage';
}

/**
* @return string
*/
public function getDescription(): string
{
return 'MinIO Object Storage';
}
}
5 changes: 3 additions & 2 deletions src/Storage/Device/S3.php
Original file line number Diff line number Diff line change
Expand Up @@ -116,6 +116,7 @@ public function __construct(string $root, string $accessKey, string $secretKey,
$this->acl = $acl;
$this->headers['host'] = $this->bucket . '.s3.'.$this->region.'.amazonaws.com';
$this->amzHeaders = [];
$this->protocol = 'https';
}

/**
Expand Down Expand Up @@ -659,9 +660,9 @@ private function getSignatureV4(string $method, string $uri, array $parameters =
*
* @return object
*/
private function call(string $method, string $uri, string $data = '', array $parameters=[])
public function call(string $method, string $uri, string $data = '', array $parameters=[])
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
public function call(string $method, string $uri, string $data = '', array $parameters=[])
private function call(string $method, string $uri, string $data = '', array $parameters=[])

Please avoid changing the access modifiers

{
$url = 'https://' . $this->headers['host'] . $uri . '?' . \http_build_query($parameters, '', '&', PHP_QUERY_RFC3986);
$url = $this->protocol . '://' . $this->headers['host'] . $uri . '?' . \http_build_query($parameters, '', '&', PHP_QUERY_RFC3986);
$response = new \stdClass;
$response->body = '';
$response->headers = [];
Expand Down
1 change: 1 addition & 0 deletions src/Storage/Storage.php
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ class Storage
const DEVICE_WASABI = 'Wasabi';
const DEVICE_BACKBLAZE = 'Backblaze';
const DEVICE_LINODE= 'Linode';
const DEVICE_MINIO= 'MinIO';

/**
* Devices.
Expand Down
32 changes: 32 additions & 0 deletions tests/Storage/Device/MinIOTest.php
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
<?php

namespace Utopia\Tests;

use Utopia\Storage\Device\MinIO;
use Utopia\Tests\S3Base;

class MinIOTest extends S3Base
{
protected function init(): void
{
$this->root = 'minio-test-bucket';
$key = $_SERVER['MINIO_ACCESS_KEY'] ?? '';
$secret = $_SERVER['MINIO_SECRET'] ?? '';
$protocol = $_SERVER['MINIO_PROTOCOL'] ?? '';
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
$protocol = $_SERVER['MINIO_PROTOCOL'] ?? '';

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Protocol has to be here, because MinIO can be hosted in an internal Network, where you don't have SSL

$host = $_SERVER['MINIO_HOST'] ?? '';
$bucket = 'minio-test-bucket';

$this->object = new MinIO($this->root, $key, $secret, $protocol, $host, $bucket, MinIO::EU_CENTRAL_1);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
$this->object = new MinIO($this->root, $key, $secret, $protocol, $host, $bucket, MinIO::EU_CENTRAL_1);
$this->object = new MinIO($this->root, $key, $secret, $host, $bucket, MinIO::EU_CENTRAL_1);

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Protocol has to be here, because MinIO can be hosted in an internal Network, where you don't have SSL


}

protected function getAdapterName(): string
{
return 'MinIO Object Storage';
}

protected function getAdapterDescription(): string
{
return 'MinIO Object Storage';
}
}