From a71f13043b918d691a71d10fcc5c3f11019935f8 Mon Sep 17 00:00:00 2001 From: Moary Chen Date: Tue, 17 Nov 2020 13:59:50 +0800 Subject: [PATCH 1/7] Add resource searching sample description --- .../README.md | 32 +++++++++++++++++++ 1 file changed, 32 insertions(+) diff --git a/sdk/spring/azure-spring-boot-starter-storage/README.md b/sdk/spring/azure-spring-boot-starter-storage/README.md index 865866f7d3083..cafb604f0d8f6 100644 --- a/sdk/spring/azure-spring-boot-starter-storage/README.md +++ b/sdk/spring/azure-spring-boot-starter-storage/README.md @@ -72,6 +72,38 @@ private final BlobServiceAsyncClient blobServiceAsyncClient = blobServiceClientB ``` +#### Search for resources +You can use implementation class `AzureStorageResourcePatternResolver` of `ResourcePatternResolver` to search resource, it supports `blob` or `file` type. +* Inject the builder bean. + + ```java + @Autowired + BlobServiceClientBuilder blobServiceClientBuilder; + + @Autowired + ShareServiceClientBuilder shareServiceClientBuilder; + ``` + +* Create pattern object, if only one storage type resource used, please call the corresponding constructor. + ```java + BlobServiceClient blobServiceClient = blobServiceClientBuilder.buildClient(); + ShareServiceClient shareServiceClient = shareServiceClientBuilder.buildClient(); + storageResourcePatternResolver = new AzureStorageResourcePatternResolver(blobServiceClient, shareServiceClient); + ``` + +* Pattern usage, the **searchPattern** should start with `azure-blob://` or `azure-file://`. Such as `azure-blob://*/*`, it means list all blobs in all containers; `azure-blob://demo-container/**`, it means list all blobs in the demo-container container, including any sub-folder. + + ```java + Resource[] resources = storageResourcePatternResolver.getResources(searchPattern); + ``` + + >**Note:** If there are too many matching files, partial data will be returned. + +* Location usage, the **searchLocation** should start with `azure-blob://` or `azure-file://`, the remaing file path should exist, otherwise an exception will be thrown. Such as `azure-blob://*/*`, it means list all blobs in all containers; `azure-blob://demo-container/**`, it means list all blobs in the demo-container container, including any sub-folder. + ```java + Resource[] resources = storageResourcePatternResolver.getResource(searchLocation); + ``` + ## Troubleshooting ### Enable client logging Azure SDKs for Java offers a consistent logging story to help aid in troubleshooting application errors and expedite their resolution. The logs produced will capture the flow of an application before reaching the terminal state to help locate the root issue. View the [logging][logging] wiki for guidance about enabling logging. From 5bd3359594616955087f61d8e2fc2ee64bb7573f Mon Sep 17 00:00:00 2001 From: Moary Chen Date: Wed, 18 Nov 2020 10:38:49 +0800 Subject: [PATCH 2/7] Combine the sample code and comment in java code block --- .../README.md | 121 ++++++++++++++---- 1 file changed, 94 insertions(+), 27 deletions(-) diff --git a/sdk/spring/azure-spring-boot-starter-storage/README.md b/sdk/spring/azure-spring-boot-starter-storage/README.md index cafb604f0d8f6..ed550f77e0fba 100644 --- a/sdk/spring/azure-spring-boot-starter-storage/README.md +++ b/sdk/spring/azure-spring-boot-starter-storage/README.md @@ -74,35 +74,102 @@ private final BlobServiceAsyncClient blobServiceAsyncClient = blobServiceClientB #### Search for resources You can use implementation class `AzureStorageResourcePatternResolver` of `ResourcePatternResolver` to search resource, it supports `blob` or `file` type. -* Inject the builder bean. +```java +import com.azure.spring.autoconfigure.storage.resource.AzureStorageResourcePatternResolver; +import com.azure.storage.blob.BlobServiceClient; +import com.azure.storage.blob.BlobServiceClientBuilder; +import com.azure.storage.file.share.ShareServiceClient; +import com.azure.storage.file.share.ShareServiceClientBuilder; +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.core.io.Resource; +import org.springframework.web.bind.annotation.GetMapping; +import org.springframework.web.bind.annotation.RequestMapping; +import org.springframework.web.bind.annotation.RequestParam; +import org.springframework.web.bind.annotation.RestController; + +import java.io.IOException; +import java.util.Arrays; + +@RestController +@RequestMapping("search") +public class BlobOrFileSearchController { - ```java - @Autowired - BlobServiceClientBuilder blobServiceClientBuilder; - @Autowired - ShareServiceClientBuilder shareServiceClientBuilder; - ``` - -* Create pattern object, if only one storage type resource used, please call the corresponding constructor. - ```java - BlobServiceClient blobServiceClient = blobServiceClientBuilder.buildClient(); - ShareServiceClient shareServiceClient = shareServiceClientBuilder.buildClient(); - storageResourcePatternResolver = new AzureStorageResourcePatternResolver(blobServiceClient, shareServiceClient); - ``` - -* Pattern usage, the **searchPattern** should start with `azure-blob://` or `azure-file://`. Such as `azure-blob://*/*`, it means list all blobs in all containers; `azure-blob://demo-container/**`, it means list all blobs in the demo-container container, including any sub-folder. - - ```java - Resource[] resources = storageResourcePatternResolver.getResources(searchPattern); - ``` - - >**Note:** If there are too many matching files, partial data will be returned. - -* Location usage, the **searchLocation** should start with `azure-blob://` or `azure-file://`, the remaing file path should exist, otherwise an exception will be thrown. Such as `azure-blob://*/*`, it means list all blobs in all containers; `azure-blob://demo-container/**`, it means list all blobs in the demo-container container, including any sub-folder. - ```java - Resource[] resources = storageResourcePatternResolver.getResource(searchLocation); - ``` + private BlobServiceClientBuilder blobServiceClientBuilder; + + @Autowired(required = false) + private ShareServiceClientBuilder shareServiceClientBuilder; + + private AzureStorageResourcePatternResolver storageResourcePatternResolver; + + @GetMapping("pattern") + public String pattern(@RequestParam String pattern){ + StringBuffer sb = new StringBuffer(); + Resource[] resources = searchResourcesByPattern(pattern); + if (null != resources) { + Arrays.stream(resources).forEach(res -> { + sb.append(res.getFilename() + "\t"); + }); + } + return sb.toString(); + } + + @GetMapping("location") + public String location(@RequestParam String location){ + StringBuffer sb = new StringBuffer(); + Resource resource = searchResourceByLocation(location); + if (null != resource) { + sb.append(resource.getFilename()); + } + return sb.toString(); + } + + /** + * Create singleton pattern resolver object, if only one storage type resource used, please call the corresponding constructor. + * @return AzureStorageResourcePatternResolver instance. + */ + private AzureStorageResourcePatternResolver getStorageResourcePatternResolver() { + if (null == storageResourcePatternResolver) { + synchronized (AzureStorageResourcePatternResolver.class) { + BlobServiceClient blobServiceClient = blobServiceClientBuilder.buildClient(); + ShareServiceClient shareServiceClient = null; + if (null != shareServiceClientBuilder) { + shareServiceClient = shareServiceClientBuilder.buildClient(); + } + storageResourcePatternResolver = new AzureStorageResourcePatternResolver(blobServiceClient, shareServiceClient); + } + } + return storageResourcePatternResolver; + } + + /** + * Pattern search, the 'searchPattern' should start with 'azure-blob://' or 'azure-file://'. + * Such as 'azure-blob://*/*', it means list all blobs in all containers + * 'azure-blob://demo-container/**', it means list all blobs in the demo-container container, including any sub-folder. + * @param searchPattern pattern expression + * @return Matching resource array + */ + public Resource[] searchResourcesByPattern(String searchPattern){ + Resource[] resources = null; + try { + resources = getStorageResourcePatternResolver().getResources(searchPattern); + } catch (IOException e) { + e.printStackTrace(); + } + return resources; + } + + /** + * Location search, the 'searchLocation' should start with 'azure-blob://' or 'azure-file://', + * the remaining file path should exist, otherwise an exception will be thrown. + * @param searchLocation file full path + * @return Query resource + */ + public Resource searchResourceByLocation(String searchLocation){ + return getStorageResourcePatternResolver().getResource(searchLocation); + } +} +``` ## Troubleshooting ### Enable client logging From 5a7654cd75278e0a9d369cf7210fb38383deb83d Mon Sep 17 00:00:00 2001 From: Moary Chen Date: Wed, 18 Nov 2020 11:02:19 +0800 Subject: [PATCH 3/7] Simple the sample code and comment --- .../README.md | 101 ++---------------- 1 file changed, 7 insertions(+), 94 deletions(-) diff --git a/sdk/spring/azure-spring-boot-starter-storage/README.md b/sdk/spring/azure-spring-boot-starter-storage/README.md index ed550f77e0fba..99a03c2460201 100644 --- a/sdk/spring/azure-spring-boot-starter-storage/README.md +++ b/sdk/spring/azure-spring-boot-starter-storage/README.md @@ -74,101 +74,14 @@ private final BlobServiceAsyncClient blobServiceAsyncClient = blobServiceClientB #### Search for resources You can use implementation class `AzureStorageResourcePatternResolver` of `ResourcePatternResolver` to search resource, it supports `blob` or `file` type. +* Create pattern resolver object, if only one storage type resource used, please call the corresponding constructor. +* Pattern search, the **searchPattern** should start with `azure-blob://` or `azure-file://`. Such as `azure-blob://*/*`, it means list all blobs in all containers; `azure-blob://demo-container/**`, it means list all blobs in the demo-container container, including any sub-folder. +* Location search, the **searchLocation** should start with `azure-blob://` or `azure-file://`, the remaining file path should exist, otherwise an exception will be thrown. + ```java -import com.azure.spring.autoconfigure.storage.resource.AzureStorageResourcePatternResolver; -import com.azure.storage.blob.BlobServiceClient; -import com.azure.storage.blob.BlobServiceClientBuilder; -import com.azure.storage.file.share.ShareServiceClient; -import com.azure.storage.file.share.ShareServiceClientBuilder; -import org.springframework.beans.factory.annotation.Autowired; -import org.springframework.core.io.Resource; -import org.springframework.web.bind.annotation.GetMapping; -import org.springframework.web.bind.annotation.RequestMapping; -import org.springframework.web.bind.annotation.RequestParam; -import org.springframework.web.bind.annotation.RestController; - -import java.io.IOException; -import java.util.Arrays; - -@RestController -@RequestMapping("search") -public class BlobOrFileSearchController { - - @Autowired - private BlobServiceClientBuilder blobServiceClientBuilder; - - @Autowired(required = false) - private ShareServiceClientBuilder shareServiceClientBuilder; - - private AzureStorageResourcePatternResolver storageResourcePatternResolver; - - @GetMapping("pattern") - public String pattern(@RequestParam String pattern){ - StringBuffer sb = new StringBuffer(); - Resource[] resources = searchResourcesByPattern(pattern); - if (null != resources) { - Arrays.stream(resources).forEach(res -> { - sb.append(res.getFilename() + "\t"); - }); - } - return sb.toString(); - } - - @GetMapping("location") - public String location(@RequestParam String location){ - StringBuffer sb = new StringBuffer(); - Resource resource = searchResourceByLocation(location); - if (null != resource) { - sb.append(resource.getFilename()); - } - return sb.toString(); - } - - /** - * Create singleton pattern resolver object, if only one storage type resource used, please call the corresponding constructor. - * @return AzureStorageResourcePatternResolver instance. - */ - private AzureStorageResourcePatternResolver getStorageResourcePatternResolver() { - if (null == storageResourcePatternResolver) { - synchronized (AzureStorageResourcePatternResolver.class) { - BlobServiceClient blobServiceClient = blobServiceClientBuilder.buildClient(); - ShareServiceClient shareServiceClient = null; - if (null != shareServiceClientBuilder) { - shareServiceClient = shareServiceClientBuilder.buildClient(); - } - storageResourcePatternResolver = new AzureStorageResourcePatternResolver(blobServiceClient, shareServiceClient); - } - } - return storageResourcePatternResolver; - } - - /** - * Pattern search, the 'searchPattern' should start with 'azure-blob://' or 'azure-file://'. - * Such as 'azure-blob://*/*', it means list all blobs in all containers - * 'azure-blob://demo-container/**', it means list all blobs in the demo-container container, including any sub-folder. - * @param searchPattern pattern expression - * @return Matching resource array - */ - public Resource[] searchResourcesByPattern(String searchPattern){ - Resource[] resources = null; - try { - resources = getStorageResourcePatternResolver().getResources(searchPattern); - } catch (IOException e) { - e.printStackTrace(); - } - return resources; - } - - /** - * Location search, the 'searchLocation' should start with 'azure-blob://' or 'azure-file://', - * the remaining file path should exist, otherwise an exception will be thrown. - * @param searchLocation file full path - * @return Query resource - */ - public Resource searchResourceByLocation(String searchLocation){ - return getStorageResourcePatternResolver().getResource(searchLocation); - } -} +AzureStorageResourcePatternResolver storageResourcePatternResolver = new AzureStorageResourcePatternResolver(blobServiceClientBuilder.buildClient()); +Resource[] resources = storageResourcePatternResolver.getResources(searchPattern); +Resource resource = storageResourcePatternResolver.getResource(searchLocation) ``` ## Troubleshooting From 17899d3e1e28dea36f978924a56265c7f809e8a5 Mon Sep 17 00:00:00 2001 From: Moary Chen Date: Wed, 18 Nov 2020 12:55:11 +0800 Subject: [PATCH 4/7] Add Multipart upload section --- .../README.md | 53 ++++++++++++++++++- 1 file changed, 52 insertions(+), 1 deletion(-) diff --git a/sdk/spring/azure-spring-boot-starter-storage/README.md b/sdk/spring/azure-spring-boot-starter-storage/README.md index 99a03c2460201..daed34274dc51 100644 --- a/sdk/spring/azure-spring-boot-starter-storage/README.md +++ b/sdk/spring/azure-spring-boot-starter-storage/README.md @@ -81,7 +81,55 @@ You can use implementation class `AzureStorageResourcePatternResolver` of `Resou ```java AzureStorageResourcePatternResolver storageResourcePatternResolver = new AzureStorageResourcePatternResolver(blobServiceClientBuilder.buildClient()); Resource[] resources = storageResourcePatternResolver.getResources(searchPattern); -Resource resource = storageResourcePatternResolver.getResource(searchLocation) +Resource resource = storageResourcePatternResolver.getResource(searchLocation); +``` + +#### Multipart upload +There are two steps: +* Upload to a block blob with `blockId`, the `blockId` must be less than or equal to 64 bytes in size, please note the Base64 string should be URL-encoded first. For a given blob, the length of the value specified for the `blockId` parameter must be the same size for each block. Learn more at [Put Block][put-block]. +* Commit the block list to save the block blob with block id list. Learn more at [Put Block List][put-block-list]. + +```java +/** + * Multiple block upload scenario. + * @throws UnsupportedEncodingException + */ +public void multipartUpload() throws UnsupportedEncodingException { + String containerName = "demo-container"; + String blobName = "demo-block-name"; + + // First block information + String blockIdFirst = getBase64BlockId("demo-block-id-001"); + String uploadDataFirst = "demo uploaded data first"; + + // Second block information + String blockIdSecond = getBase64BlockId("demo-block-id-002"); + String uploadDataSecond = "demo uploaded data second"; + + // Construct BlockBlobClient instance + BlobServiceClient blobServiceClient = blobServiceClientBuilder.buildClient(); + BlobContainerClient blobContainerClient = blobServiceClient.getBlobContainerClient(containerName); + BlockBlobClient blockBlobClient = blobContainerClient.getBlobClient(blobName).getBlockBlobClient(); + + // Upload difference blocks + blockBlobClient.stageBlock(blockIdFirst, new ByteArrayInputStream(uploadDataFirst.getBytes()), uploadDataFirst.length()); + blockBlobClient.stageBlock(blockIdSecond, new ByteArrayInputStream(uploadDataSecond.getBytes()), uploadDataSecond.length()); + + // Commit block list + blockBlobClient.commitBlockList(Arrays.asList(blockIdFirst, blockIdSecond)); +} + +/** + * Encode block id, use URLEncoder encode first, then use Base64 encode + * @param blockId customized block id. + * @return encoded block id + * @throws UnsupportedEncodingException + */ +private String getBase64BlockId(String blockId) throws UnsupportedEncodingException { + return Base64.getEncoder() + .encodeToString(URLEncoder.encode(blockId, StandardCharsets.UTF_8.toString()) + .getBytes()); +} ``` ## Troubleshooting @@ -127,3 +175,6 @@ Please follow [instructions here][contributing_md] to build from source or contr [other_operation]: https://docs.spring.io/spring/docs/current/spring-framework-reference/core.html#resources [jdk_link]: https://docs.microsoft.com/java/azure/jdk/?view=azure-java-stable +[put-block]: https://docs.microsoft.com/rest/api/storageservices/put-block +[put-block-list]: https://docs.microsoft.com/rest/api/storageservices/put-block-list + From 5647a1dc76163b97d1e0b826bff22d07ded11cde Mon Sep 17 00:00:00 2001 From: Moary Chen Date: Thu, 19 Nov 2020 11:48:48 +0800 Subject: [PATCH 5/7] Update multipart upload description --- .../README.md | 50 +------------------ 1 file changed, 2 insertions(+), 48 deletions(-) diff --git a/sdk/spring/azure-spring-boot-starter-storage/README.md b/sdk/spring/azure-spring-boot-starter-storage/README.md index daed34274dc51..ebc569e1f1c31 100644 --- a/sdk/spring/azure-spring-boot-starter-storage/README.md +++ b/sdk/spring/azure-spring-boot-starter-storage/README.md @@ -85,52 +85,7 @@ Resource resource = storageResourcePatternResolver.getResource(searchLocation); ``` #### Multipart upload -There are two steps: -* Upload to a block blob with `blockId`, the `blockId` must be less than or equal to 64 bytes in size, please note the Base64 string should be URL-encoded first. For a given blob, the length of the value specified for the `blockId` parameter must be the same size for each block. Learn more at [Put Block][put-block]. -* Commit the block list to save the block blob with block id list. Learn more at [Put Block List][put-block-list]. - -```java -/** - * Multiple block upload scenario. - * @throws UnsupportedEncodingException - */ -public void multipartUpload() throws UnsupportedEncodingException { - String containerName = "demo-container"; - String blobName = "demo-block-name"; - - // First block information - String blockIdFirst = getBase64BlockId("demo-block-id-001"); - String uploadDataFirst = "demo uploaded data first"; - - // Second block information - String blockIdSecond = getBase64BlockId("demo-block-id-002"); - String uploadDataSecond = "demo uploaded data second"; - - // Construct BlockBlobClient instance - BlobServiceClient blobServiceClient = blobServiceClientBuilder.buildClient(); - BlobContainerClient blobContainerClient = blobServiceClient.getBlobContainerClient(containerName); - BlockBlobClient blockBlobClient = blobContainerClient.getBlobClient(blobName).getBlockBlobClient(); - - // Upload difference blocks - blockBlobClient.stageBlock(blockIdFirst, new ByteArrayInputStream(uploadDataFirst.getBytes()), uploadDataFirst.length()); - blockBlobClient.stageBlock(blockIdSecond, new ByteArrayInputStream(uploadDataSecond.getBytes()), uploadDataSecond.length()); - - // Commit block list - blockBlobClient.commitBlockList(Arrays.asList(blockIdFirst, blockIdSecond)); -} - -/** - * Encode block id, use URLEncoder encode first, then use Base64 encode - * @param blockId customized block id. - * @return encoded block id - * @throws UnsupportedEncodingException - */ -private String getBase64BlockId(String blockId) throws UnsupportedEncodingException { - return Base64.getEncoder() - .encodeToString(URLEncoder.encode(blockId, StandardCharsets.UTF_8.toString()) - .getBytes()); -} -``` +Files larger than 4 MiB will be uploaded to Azure Storage in parallel. Learn more at [Upload in parallel][storage-blob-scalable-app-upload-files]. ## Troubleshooting ### Enable client logging @@ -175,6 +130,5 @@ Please follow [instructions here][contributing_md] to build from source or contr [other_operation]: https://docs.spring.io/spring/docs/current/spring-framework-reference/core.html#resources [jdk_link]: https://docs.microsoft.com/java/azure/jdk/?view=azure-java-stable -[put-block]: https://docs.microsoft.com/rest/api/storageservices/put-block -[put-block-list]: https://docs.microsoft.com/rest/api/storageservices/put-block-list +[storage-blob-scalable-app-upload-files]: https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blob-scalable-app-upload-files From 2c0ab676623a38c7e458d5112b2c6b0831d6feb5 Mon Sep 17 00:00:00 2001 From: Moary Chen Date: Thu, 19 Nov 2020 11:50:18 +0800 Subject: [PATCH 6/7] Update link --- sdk/spring/azure-spring-boot-starter-storage/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sdk/spring/azure-spring-boot-starter-storage/README.md b/sdk/spring/azure-spring-boot-starter-storage/README.md index ebc569e1f1c31..51a293a4e7c21 100644 --- a/sdk/spring/azure-spring-boot-starter-storage/README.md +++ b/sdk/spring/azure-spring-boot-starter-storage/README.md @@ -130,5 +130,5 @@ Please follow [instructions here][contributing_md] to build from source or contr [other_operation]: https://docs.spring.io/spring/docs/current/spring-framework-reference/core.html#resources [jdk_link]: https://docs.microsoft.com/java/azure/jdk/?view=azure-java-stable -[storage-blob-scalable-app-upload-files]: https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blob-scalable-app-upload-files +[storage-blob-scalable-app-upload-files]: https://docs.microsoft.com/azure/storage/blobs/storage-blob-scalable-app-upload-files From fe52fe40fc6a118a413b58c3f7482f408ce04050 Mon Sep 17 00:00:00 2001 From: Moary Chen Date: Thu, 19 Nov 2020 14:00:41 +0800 Subject: [PATCH 7/7] revised according to review comments --- sdk/spring/azure-spring-boot-starter-storage/README.md | 7 ++----- 1 file changed, 2 insertions(+), 5 deletions(-) diff --git a/sdk/spring/azure-spring-boot-starter-storage/README.md b/sdk/spring/azure-spring-boot-starter-storage/README.md index 51a293a4e7c21..269bc1953cc12 100644 --- a/sdk/spring/azure-spring-boot-starter-storage/README.md +++ b/sdk/spring/azure-spring-boot-starter-storage/README.md @@ -74,18 +74,18 @@ private final BlobServiceAsyncClient blobServiceAsyncClient = blobServiceClientB #### Search for resources You can use implementation class `AzureStorageResourcePatternResolver` of `ResourcePatternResolver` to search resource, it supports `blob` or `file` type. -* Create pattern resolver object, if only one storage type resource used, please call the corresponding constructor. * Pattern search, the **searchPattern** should start with `azure-blob://` or `azure-file://`. Such as `azure-blob://*/*`, it means list all blobs in all containers; `azure-blob://demo-container/**`, it means list all blobs in the demo-container container, including any sub-folder. * Location search, the **searchLocation** should start with `azure-blob://` or `azure-file://`, the remaining file path should exist, otherwise an exception will be thrown. ```java AzureStorageResourcePatternResolver storageResourcePatternResolver = new AzureStorageResourcePatternResolver(blobServiceClientBuilder.buildClient()); + Resource[] resources = storageResourcePatternResolver.getResources(searchPattern); Resource resource = storageResourcePatternResolver.getResource(searchLocation); ``` #### Multipart upload -Files larger than 4 MiB will be uploaded to Azure Storage in parallel. Learn more at [Upload in parallel][storage-blob-scalable-app-upload-files]. +Files larger than 4 MiB will be uploaded to Azure Storage in parallel. ## Troubleshooting ### Enable client logging @@ -129,6 +129,3 @@ Please follow [instructions here][contributing_md] to build from source or contr [azure_storage]: https://azure.microsoft.com/services/storage/blobs/ [other_operation]: https://docs.spring.io/spring/docs/current/spring-framework-reference/core.html#resources [jdk_link]: https://docs.microsoft.com/java/azure/jdk/?view=azure-java-stable - -[storage-blob-scalable-app-upload-files]: https://docs.microsoft.com/azure/storage/blobs/storage-blob-scalable-app-upload-files -