[metricsadvisor] live tests setup #14396

kristapratico · 2020-10-09T18:27:32Z

Once checked into master, will run pipeline generator and make any adjustments necessary in another PR

xiangyan99 · 2020-10-12T19:58:13Z

sdk/metricsadvisor/tests.yml

+        AZURE_CLIENT_SECRET: $(aad-azure-sdk-test-client-secret)
+        AZURE_CLIENT_ID: $(aad-azure-sdk-test-client-id)
+        AZURE_METRICS_ADVISOR_ENDPOINT: $(metricsadvisor-test-service-endpoint)
+        AZURE_METRICS_ADVISOR_SUBSCRIPTION_KEY: $(metricsadvisor-test-subscription-key)


Do we share these variables with other language?

Yes, all environment variables are shared across languages except API key

xiangyan99 · 2020-10-12T19:58:34Z

sdk/metricsadvisor/azure-ai-metricsadvisor/tests/async_tests/base_testcase_async.py

-            self.alert_id = self.get_settings_value("ALERT_ID")
+            self.anomaly_detection_configuration_id = "c0f2539f-b804-4ab9-a70f-0da0c89c76d8"
+            self.anomaly_alert_configuration_id = "ff3014a0-bbbb-41ec-a637-677e77b81299"
+            self.metric_id = "3d48ed3e-6e6e-4391-b78f-b00dfee1e6f5"


Why we hard code the id here?

So we don't have to add them to the test secrets keyvault.

But if we modify the secrets that we saved in keyvault, we have to update them as well, right?

I mean these ids are coupled with other secrets.

yeah that's a good point. We were trying to minimize the number of things we were adding to the keyvault, but I think it probably makes more sense to allow ease of update here. I'll add them back as env vars.

* [metricsadvisor] live tests setup (#14396) * add live test yml for metrics advisor * some fixes * hardcode test ids and rerecord a few tests * add ids back as env variables * [formrecognizer] initial business cards (#14026) * Sync eng/common directory with azure-sdk-tools for PR 1081 (#14445) Sync eng/common directory with azure-sdk-tools for PR Azure/azure-sdk-tools#1081 See [eng/common workflow](https://github.com/Azure/azure-sdk-tools/blob/master/eng/common/README.md#workflow) * Communication Administration: Add on-demand resource creation for live-tests (#14366) * Add communication service preparer * Refactor dynamic resource creation testing code * Remove setup method of base test class * Change fake conn str to valid format * Remove unused import * Remove main test file * Reduce RG expiry durtion * Async test code change with resource preparer * Add chaching for resource-preparer * Move helper into shared * Move preparer into shared * Uncomment test code * Remove CommunicationResourceGroupPreparer use common instead * Add base testcase for sync pnm * move phone_number_helper to phone_number folder * Add base pnm async testcase * Add __init__ for test folders * Fix identity tests to refer to shared test folder * Refactor base async phonenumber test class * Make decorator consisitent across all async tests utm * Replace fake resource value * Fix type in test file names * Remove commented cache setting code * Add livetest recording files * Refresh recording files * Update recording files * Reorganize the test folder structure * Add mgmt pacakge to the dev reqs Co-authored-by: tural farhadov <[email protected]> * [ServiceBus] Small stress test and sample touchups (#14304) * Emplace sample generated for customer issue on the side for future reference. (generate sas token and authenticate) * Adjust stress test that was using push receive instead of pull. * Add async version of non-session to session send failure test, make both cached to improve perf. * Codegen for Azure Monitor (#14057) * Codegen fro Azure Monitor * codeowner * commit 2 * Update sdk/monitor/opentelemetry-exporter-azuremonitor/README.md * doc setings * Update eng/.docsettings.yml * Update eng/.docsettings.yml * Update sdk/monitor/opentelemetry-exporter-azuremonitor/README.md * Update eng/.docsettings.yml * version * oops * comments * Update sdk/monitor/opentelemetry-exporter-azuremonitor/LICENSE.txt Co-authored-by: Hector Hernandez <[email protected]> Co-authored-by: Hector Hernandez <[email protected]> * [Storage][FileShare]Regenerate for delete leased snapshot (#14469) * add repr to models (#14475) * remove validations (#14476) * Ma remove readonly assignments (#14478) * Ensure azure-identity readme has exactly one H1 (#14480) The docs publishing system disallows multiple H1 headings in a document. * Sync eng/common directory with azure-sdk-tools for PR 1088 (#14482) * Add pipeline configuration for cleaning up upstream branches * Fix powershell naming * Switch to use github API * Use github API to delete branches Co-authored-by: Chidozie Ononiwu <[email protected]> * enable live tests (#14467) * Removed asynctestcase.py (#14444) * testcase.py now uses AzureTestCase (#14443) Co-authored-by: Krista Pratico <[email protected]> Co-authored-by: Azure SDK Bot <[email protected]> Co-authored-by: turalf <[email protected]> Co-authored-by: tural farhadov <[email protected]> Co-authored-by: KieranBrantnerMagee <[email protected]> Co-authored-by: Rakshith Bhyravabhotla <[email protected]> Co-authored-by: Hector Hernandez <[email protected]> Co-authored-by: Xiaoxi Fu <[email protected]> Co-authored-by: Charles Lowell <[email protected]> Co-authored-by: Chidozie Ononiwu <[email protected]> Co-authored-by: Bradley D'Amato <[email protected]>

* customize httpadapter to honor kwargs in send method * update * update * update * update * update * update * update * FI from master (#14501) * [metricsadvisor] live tests setup (#14396) * add live test yml for metrics advisor * some fixes * hardcode test ids and rerecord a few tests * add ids back as env variables * [formrecognizer] initial business cards (#14026) * Sync eng/common directory with azure-sdk-tools for PR 1081 (#14445) Sync eng/common directory with azure-sdk-tools for PR Azure/azure-sdk-tools#1081 See [eng/common workflow](https://github.com/Azure/azure-sdk-tools/blob/master/eng/common/README.md#workflow) * Communication Administration: Add on-demand resource creation for live-tests (#14366) * Add communication service preparer * Refactor dynamic resource creation testing code * Remove setup method of base test class * Change fake conn str to valid format * Remove unused import * Remove main test file * Reduce RG expiry durtion * Async test code change with resource preparer * Add chaching for resource-preparer * Move helper into shared * Move preparer into shared * Uncomment test code * Remove CommunicationResourceGroupPreparer use common instead * Add base testcase for sync pnm * move phone_number_helper to phone_number folder * Add base pnm async testcase * Add __init__ for test folders * Fix identity tests to refer to shared test folder * Refactor base async phonenumber test class * Make decorator consisitent across all async tests utm * Replace fake resource value * Fix type in test file names * Remove commented cache setting code * Add livetest recording files * Refresh recording files * Update recording files * Reorganize the test folder structure * Add mgmt pacakge to the dev reqs Co-authored-by: tural farhadov <[email protected]> * [ServiceBus] Small stress test and sample touchups (#14304) * Emplace sample generated for customer issue on the side for future reference. (generate sas token and authenticate) * Adjust stress test that was using push receive instead of pull. * Add async version of non-session to session send failure test, make both cached to improve perf. * Codegen for Azure Monitor (#14057) * Codegen fro Azure Monitor * codeowner * commit 2 * Update sdk/monitor/opentelemetry-exporter-azuremonitor/README.md * doc setings * Update eng/.docsettings.yml * Update eng/.docsettings.yml * Update sdk/monitor/opentelemetry-exporter-azuremonitor/README.md * Update eng/.docsettings.yml * version * oops * comments * Update sdk/monitor/opentelemetry-exporter-azuremonitor/LICENSE.txt Co-authored-by: Hector Hernandez <[email protected]> Co-authored-by: Hector Hernandez <[email protected]> * [Storage][FileShare]Regenerate for delete leased snapshot (#14469) * add repr to models (#14475) * remove validations (#14476) * Ma remove readonly assignments (#14478) * Ensure azure-identity readme has exactly one H1 (#14480) The docs publishing system disallows multiple H1 headings in a document. * Sync eng/common directory with azure-sdk-tools for PR 1088 (#14482) * Add pipeline configuration for cleaning up upstream branches * Fix powershell naming * Switch to use github API * Use github API to delete branches Co-authored-by: Chidozie Ononiwu <[email protected]> * enable live tests (#14467) * Removed asynctestcase.py (#14444) * testcase.py now uses AzureTestCase (#14443) Co-authored-by: Krista Pratico <[email protected]> Co-authored-by: Azure SDK Bot <[email protected]> Co-authored-by: turalf <[email protected]> Co-authored-by: tural farhadov <[email protected]> Co-authored-by: KieranBrantnerMagee <[email protected]> Co-authored-by: Rakshith Bhyravabhotla <[email protected]> Co-authored-by: Hector Hernandez <[email protected]> Co-authored-by: Xiaoxi Fu <[email protected]> Co-authored-by: Charles Lowell <[email protected]> Co-authored-by: Chidozie Ononiwu <[email protected]> Co-authored-by: Bradley D'Amato <[email protected]> * update * update * update * update * update * update * update * update Co-authored-by: Krista Pratico <[email protected]> Co-authored-by: Azure SDK Bot <[email protected]> Co-authored-by: turalf <[email protected]> Co-authored-by: tural farhadov <[email protected]> Co-authored-by: KieranBrantnerMagee <[email protected]> Co-authored-by: Rakshith Bhyravabhotla <[email protected]> Co-authored-by: Hector Hernandez <[email protected]> Co-authored-by: Xiaoxi Fu <[email protected]> Co-authored-by: Charles Lowell <[email protected]> Co-authored-by: Chidozie Ononiwu <[email protected]> Co-authored-by: Bradley D'Amato <[email protected]>

add error response flag (Azure#14396)

add live test yml for metrics advisor

4597d94

kristapratico added the Cognitive - Metrics Advisor label Oct 9, 2020

kristapratico self-assigned this Oct 9, 2020

kristapratico added 2 commits October 9, 2020 13:23

some fixes

4213c03

hardcode test ids and rerecord a few tests

0e6640f

kristapratico marked this pull request as ready for review October 12, 2020 19:40

kristapratico requested review from danieljurek and xiangyan99 as code owners October 12, 2020 19:40

xiangyan99 reviewed Oct 12, 2020

View reviewed changes

add ids back as env variables

2920148

kristapratico requested a review from xiangyan99 October 12, 2020 20:49

xiangyan99 approved these changes Oct 12, 2020

View reviewed changes

kristapratico merged commit 5eced3f into master Oct 12, 2020

kristapratico deleted the ma-live-tests branch October 12, 2020 21:44

openapi-sdkautomation bot pushed a commit to AzureSDKAutomation/azure-sdk-for-python that referenced this pull request May 14, 2021

CodeGen from PR 14396 in Azure/azure-rest-api-specs

ab39f93

add error response flag (Azure#14396)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[metricsadvisor] live tests setup #14396

[metricsadvisor] live tests setup #14396

kristapratico commented Oct 9, 2020 •

edited

Loading

xiangyan99 Oct 12, 2020

kristapratico Oct 12, 2020

xiangyan99 Oct 12, 2020

kristapratico Oct 12, 2020

xiangyan99 Oct 12, 2020

kristapratico Oct 12, 2020

[metricsadvisor] live tests setup #14396

[metricsadvisor] live tests setup #14396

Conversation

kristapratico commented Oct 9, 2020 • edited Loading

xiangyan99 Oct 12, 2020

Choose a reason for hiding this comment

kristapratico Oct 12, 2020

Choose a reason for hiding this comment

xiangyan99 Oct 12, 2020

Choose a reason for hiding this comment

kristapratico Oct 12, 2020

Choose a reason for hiding this comment

xiangyan99 Oct 12, 2020

Choose a reason for hiding this comment

kristapratico Oct 12, 2020

Choose a reason for hiding this comment

kristapratico commented Oct 9, 2020 •

edited

Loading