Add SAS Token Authentication Support to Azure Repo Plugin #42982

original-brownbear · 2019-06-07T11:57:26Z

Added setting for SAS token
Added support for the token in tests
- I have manually verified that the setting works with real Azure. I'm not 100% happy with the way I'm passing the SAS token setting to tests here, but I figured this might be the easiest approach for infra. The implemented approach allows to only test either a client with a key set or a client with set SAS token in a single Gradle run. This seemed less cumbersome than allowing to provide two separate sets of credentials and then adding code to Gradle to run the third party test goals twice. Also, for infra this is a reasonable approach I think because they can just copy the existing Jenkins job, remove the key setting and add an SAS token setting in its place in the copy => no effort required there outside of copying the Gradle target.
Relates Add shared access signature authentication support #42117

* Added setting for SAS token * Added support for the token in tests * Relates elastic#42117

elasticmachine · 2019-06-07T11:57:28Z

Pinging @elastic/es-distributed

ywelsch

I've left two smaller asks. In general, I think we should move away from the low-level connection string creation and use the objects provided by the SDK for that. We can do that as a follow-up, however, as that will possibly complicate backporting

ywelsch · 2019-06-11T07:30:11Z

...epository-azure/src/main/java/org/elasticsearch/repositories/azure/AzureStorageSettings.java

@@ -53,6 +54,10 @@
    public static final AffixSetting<SecureString> KEY_SETTING = Setting.affixKeySetting(AZURE_CLIENT_PREFIX_KEY, "key",
        key -> SecureSetting.secureString(key, null));

+    /** Azure SAS token */
+    public static final AffixSetting<SecureString> SAS_TOKEN_SETTING = Setting.affixKeySetting(AZURE_CLIENT_PREFIX_KEY, "sas_token",


can you add docs for the new setting as well?

Done in b7045d0 :)

ywelsch · 2019-06-11T07:37:31Z

...epository-azure/src/main/java/org/elasticsearch/repositories/azure/AzureStorageSettings.java

@@ -166,7 +184,6 @@ public LocationMode getLocationMode() {
    public String toString() {
        final StringBuilder sb = new StringBuilder("AzureStorageSettings{");
        sb.append("account='").append(account).append('\'');
-        sb.append(", key='").append(key).append('\'');


add connectString instead here?

@ywelsch this was actually an on purpose removal, do we really want to add secure credentials in toString methods?

I think it's better to remove the key and maybe add a "use sas token=true|false"

Unfortunately, for that we'd have to keep some field that holds this information. Maybe not worth it?

…auth

original-brownbear · 2019-06-11T07:56:44Z

@ywelsch all addressed :)

ywelsch

LGTM. I would like a second pair of eyes on this (@tlrx or @andrershov) given the backport to 6.8.

tlrx

It looks great! I left some minor comments that need to be addressed before merging. I tested it both manually and using the 3rd party tests and it worked.

There's an interesting point about adding multiple CI jobs for 3rd party tests vs. adding more gradle test tasks. I think it make sense here to add another CI job (and this is maybe what we'd like to do for S3 too).

tlrx · 2019-06-11T08:18:46Z

docs/plugins/repository-azure.asciidoc

@@ -29,14 +30,14 @@ Note that you can also define more than one account:
 bin/elasticsearch-keystore add azure.client.default.account
 bin/elasticsearch-keystore add azure.client.default.key
 bin/elasticsearch-keystore add azure.client.secondary.account
-bin/elasticsearch-keystore add azure.client.secondary.key
+bin/elasticsearch-keystore add azure.client.default.sas_token


Maybe we could have the default account settings with the secret key and the secondary account configured with SAS (and not interleaving both)?

Ah my bad that was just a mistake in copy and pasting, thanks for spotting :)

tlrx · 2019-06-11T08:19:40Z

plugins/repository-azure/build.gradle

@@ -85,10 +86,11 @@ task thirdPartyTest(type: Test) {
  include '**/AzureStorageCleanupThirdPartyTests.class'
  systemProperty 'test.azure.account', azureAccount ? azureAccount : ""
  systemProperty 'test.azure.key', azureKey ? azureKey : ""
+  systemProperty 'test.azure.sas', azureSasToken ? azureSasToken : ""


Nit: can we also use sas_token for the system property?

tlrx · 2019-06-11T08:25:18Z

...epository-azure/src/main/java/org/elasticsearch/repositories/azure/AzureStorageSettings.java

@@ -166,7 +184,6 @@ public LocationMode getLocationMode() {
    public String toString() {
        final StringBuilder sb = new StringBuilder("AzureStorageSettings{");
        sb.append("account='").append(account).append('\'');
-        sb.append(", key='").append(key).append('\'');


I think it's better to remove the key and maybe add a "use sas token=true|false"

tlrx · 2019-06-11T08:37:34Z

plugins/repository-azure/qa/microsoft-azure-storage/build.gradle

@@ -29,12 +29,14 @@ String azureAccount = System.getenv("azure_storage_account")
 String azureKey = System.getenv("azure_storage_key")
 String azureContainer = System.getenv("azure_storage_container")
 String azureBasePath = System.getenv("azure_storage_base_path")
+String azureSasToken = System.getenv("azure_storage_sas_token")

 if (!azureAccount && !azureKey && !azureContainer && !azureBasePath) {


I think we need to change this too, in order to accommodate with the 3 situations we now have: no sys props, sys prop with key, sys prop with sas token

tlrx · 2019-06-11T08:38:15Z

plugins/repository-azure/qa/microsoft-azure-storage/build.gradle

+        println "Using access key in external service tests."
+        keystore 'azure.client.integration_test.key', azureKey
+    }
+    if (azureSasToken.isEmpty() == false) {


See previous comment, CI might break here as azureSasToken will be just null?

Right ... obviously in my local testing I just set the env vars to "" when switching between auth methods :D and didn't notice.

tlrx · 2019-06-11T09:54:10Z

docs/plugins/repository-azure.asciidoc

@@ -19,7 +19,8 @@ bin/elasticsearch-keystore add azure.client.default.account
 bin/elasticsearch-keystore add azure.client.default.key
 ----------------------------------------------------------------

-Where `account` is the azure account name and `key` the azure secret key.
+Where `account` is the azure account name and `key` the azure secret key. Instead of an azure secret key under `key`, you can alternatively
+define a shared access signatures (SAS) token under `sas_token` to use for authentication instead.


I think we should say a word about the type of SAS token and the required permissions

I think the type doesn't matter just the permissions (service as well as accounts SAS should work)? I added a comment on the permissions now :)

Talked to Armin using another channel. I'd expect to see some create or list permissions too here

Alright, updated this with all the details now. create we don't need since it's implied by write for the operations we execute https://docs.microsoft.com/en-us/rest/api/storageservices/Constructing-an-Account-SAS?redirectedfrom=MSDN#blob-service (as a matter of fact it is for all operations on the blob service). But we needed list.
Also, I added some detail on the service and resource type the token has to apply to.

…auth

original-brownbear · 2019-06-11T10:36:45Z

Thanks @tlrx all points addressed I think/hope :)

tlrx

LGTM, thanks!

original-brownbear · 2019-06-11T13:49:53Z

thanks @tlrx !

original-brownbear · 2019-06-11T14:28:15Z

Just FYI, I raised an infra issue for getting the new CI job going.

) * Added setting for SAS token * Added support for the token in tests * Relates elastic#42117

…43618) * Added setting for SAS token * Added support for the token in tests * Relates #42117

) (elastic#43618) * Added setting for SAS token * Added support for the token in tests * Relates elastic#42117

* Add SAS Token Authentication Support to Azure Repo Plugin (#42982) (#43618) * Added setting for SAS token * Added support for the token in tests * Relates #42117

This adds sas-token support for azure repositories based on elastic/elasticsearch#42982.

Add SAS Token Authentication Support to Azure Repo Plugin

26c514c

* Added setting for SAS token * Added support for the token in tests * Relates elastic#42117

original-brownbear added >enhancement :Distributed Coordination/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs v8.0.0 v7.3.0 v6.8.1 labels Jun 7, 2019

original-brownbear mentioned this pull request Jun 7, 2019

Add shared access signature authentication support #42117

Closed

original-brownbear added v7.1.2 v7.2.0 labels Jun 7, 2019

original-brownbear requested review from ywelsch and andrershov June 7, 2019 14:14

ywelsch requested a review from tlrx June 11, 2019 07:25

ywelsch suggested changes Jun 11, 2019

View reviewed changes

original-brownbear added 2 commits June 11, 2019 09:50

Merge remote-tracking branch 'elastic/master' into azure-service-key-…

0b6920a

…auth

CR: add docs for sas_token

b7045d0

original-brownbear requested a review from ywelsch June 11, 2019 07:56

ywelsch approved these changes Jun 11, 2019

View reviewed changes

tlrx requested changes Jun 11, 2019

View reviewed changes

original-brownbear added 2 commits June 11, 2019 12:20

Merge remote-tracking branch 'elastic/master' into azure-service-key-…

1c5d669

…auth

CR: comments

3df99c2

original-brownbear requested a review from tlrx June 11, 2019 10:36

CR: more detail permissions

1b9c905

tlrx approved these changes Jun 11, 2019

View reviewed changes

original-brownbear merged commit faffaa9 into elastic:master Jun 11, 2019

original-brownbear deleted the azure-service-key-auth branch June 11, 2019 13:50

original-brownbear added the backport pending label Jun 11, 2019

jakelandis added v6.8.2 and removed v7.2.0 v6.8.1 labels Jun 17, 2019

ywelsch removed the v7.1.2 label Jun 17, 2019

original-brownbear added a commit to original-brownbear/elasticsearch that referenced this pull request Jun 26, 2019

Add SAS Token Authentication Support to Azure Repo Plugin (elastic#42982

58f5210

) * Added setting for SAS token * Added support for the token in tests * Relates elastic#42117

original-brownbear mentioned this pull request Jun 26, 2019

Add SAS Token Authentication Support to Azure Repo Plugin (#42982) #43618

Merged

original-brownbear added a commit that referenced this pull request Jun 26, 2019

Add SAS Token Authentication Support to Azure Repo Plugin (#42982) (#…

8306796

…43618) * Added setting for SAS token * Added support for the token in tests * Relates #42117

original-brownbear mentioned this pull request Jul 1, 2019

Add SAS Token Authentication Support to Azure Repo Plugin #43841

Merged

original-brownbear removed the backport pending label Jul 1, 2019

mark-vieira mentioned this pull request Jul 23, 2019

:plugins:repository-azure:qa:microsoft-azure-storage failures #44740

Closed

Mpdreamz mentioned this pull request Aug 7, 2019

[meta] 7.3 Release elastic/elasticsearch-net#4001

Closed

16 tasks

lcawl mentioned this pull request Sep 5, 2019

[7.2][DOCS] Identify reloadable Azure repository plugin settings (#46358) #46413

Merged

jakelandis added v8.0.0-alpha1 and removed v8.0.0 labels Jul 26, 2021

proddata mentioned this pull request Jun 20, 2024

Support sas_token for Azure Snapshot repository crate/crate#16206

Closed

mkleen mentioned this pull request Jul 3, 2024

Add sas-token support for azure repositories crate/crate#16274

Merged

5 tasks

mkleen added a commit to crate/crate that referenced this pull request Jul 4, 2024

Add SAS Token support for authentification for azure repositories

372abc9

This adds sas-token support for azure repositories based on elastic/elasticsearch#42982.

mkleen added a commit to crate/crate that referenced this pull request Jul 4, 2024

Add SAS Token support for authentification for azure repositories

ed02723

This adds sas-token support for azure repositories based on elastic/elasticsearch#42982.

mkleen added a commit to crate/crate that referenced this pull request Jul 4, 2024

Add SAS Token support for authentification for azure repositories

720e53d

This adds sas-token support for azure repositories based on elastic/elasticsearch#42982.

mkleen added a commit to crate/crate that referenced this pull request Jul 4, 2024

Add SAS Token support for authentification for azure repositories

ca4d3eb

This adds sas-token support for azure repositories based on elastic/elasticsearch#42982.

mkleen added a commit to crate/crate that referenced this pull request Jul 4, 2024

Add SAS Token support for authentification for azure repositories

56f590d

This adds sas-token support for azure repositories based on elastic/elasticsearch#42982.

mkleen added a commit to crate/crate that referenced this pull request Jul 4, 2024

Add SAS Token support for authentification for azure repositories

a8e2d65

This adds sas-token support for azure repositories based on elastic/elasticsearch#42982.

mkleen added a commit to crate/crate that referenced this pull request Jul 4, 2024

Add SAS Token support for authentification for azure repositories

2b7a431

This adds sas-token support for azure repositories based on elastic/elasticsearch#42982.

mkleen added a commit to crate/crate that referenced this pull request Jul 4, 2024

Add SAS Token support for authentification for azure repositories

77d2cf7

This adds sas-token support for azure repositories based on elastic/elasticsearch#42982.

mkleen added a commit to crate/crate that referenced this pull request Jul 4, 2024

Add SAS Token support for authentification for azure repositories

02e9967

This adds sas-token support for azure repositories based on elastic/elasticsearch#42982.

mergify bot pushed a commit to crate/crate that referenced this pull request Jul 4, 2024

Add SAS Token support for authentification for azure repositories

e086900

This adds sas-token support for azure repositories based on elastic/elasticsearch#42982.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add SAS Token Authentication Support to Azure Repo Plugin #42982

Add SAS Token Authentication Support to Azure Repo Plugin #42982

original-brownbear commented Jun 7, 2019

elasticmachine commented Jun 7, 2019

ywelsch left a comment

ywelsch Jun 11, 2019

original-brownbear Jun 11, 2019

ywelsch Jun 11, 2019

original-brownbear Jun 11, 2019

tlrx Jun 11, 2019

original-brownbear Jun 11, 2019

tlrx Jun 11, 2019

original-brownbear commented Jun 11, 2019

ywelsch left a comment

tlrx left a comment

tlrx Jun 11, 2019

original-brownbear Jun 11, 2019

tlrx Jun 11, 2019

original-brownbear Jun 11, 2019

tlrx Jun 11, 2019

tlrx Jun 11, 2019

tlrx Jun 11, 2019

original-brownbear Jun 11, 2019

tlrx Jun 11, 2019

original-brownbear Jun 11, 2019

tlrx Jun 11, 2019

original-brownbear Jun 11, 2019

original-brownbear commented Jun 11, 2019

tlrx left a comment

original-brownbear commented Jun 11, 2019

original-brownbear commented Jun 11, 2019

Add SAS Token Authentication Support to Azure Repo Plugin #42982

Add SAS Token Authentication Support to Azure Repo Plugin #42982

Conversation

original-brownbear commented Jun 7, 2019

elasticmachine commented Jun 7, 2019

ywelsch left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

original-brownbear commented Jun 11, 2019

ywelsch left a comment

Choose a reason for hiding this comment

tlrx left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

original-brownbear commented Jun 11, 2019

tlrx left a comment

Choose a reason for hiding this comment

original-brownbear commented Jun 11, 2019

original-brownbear commented Jun 11, 2019