Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Core] Set default polling time to zero when running on Playback mode #18941

Closed
wants to merge 4 commits into from
Closed

[Core] Set default polling time to zero when running on Playback mode #18941

wants to merge 4 commits into from

Conversation

kinelski
Copy link
Member

@kinelski kinelski commented Feb 23, 2021 •
edited
Loading

Libraries that clearly improved Playback running time when setting default polling time to zero:

  • FormRecognizer (25.1m => 31.9s): polling time was already being manually set to zero in every test. Manual setup has been removed.
  • KeyVault.Certificates (30.2s => 950ms)

The following libraries didn't show major improvements, possibly because their APIs don't rely that much on LROs. Their running times are considerably short anyway (<10s, except for Storage):

  • Communication.PhoneNumbers
  • KeyVault.Administration
  • KeyVault.Keys
  • KeyVault.Secrets
  • Storage.Blobs
  • Synapse.Spark
  • TextAnalytics

Some libraries set their own default polling time interval, so our current approach won't work against them (follow-up: #18958).

All runs performed locally (single-thread).

Fixes #12815.

@kinelski kinelski added Client This issue points to a problem in the data-plane of the library. Azure.Core labels Feb 23, 2021
@kinelski kinelski self-assigned this Feb 23, 2021
kinelski
kinelski commented Feb 23, 2021
View reviewed changes
sdk/core/Azure.Core.TestFramework/src/RecordedTestBase.cs
{
if (Mode == RecordedTestMode.Playback)
{
foreach (var assembly in AppDomain.CurrentDomain.GetAssemblies())
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We wouldn't need the assembly lookup if the OperationHelpers class was declared as part of the Core library (instead of being manually added as a shared resource). Is there any particular reason for making it the way it is?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It wasn't an API that we wanted to publicly expose to customers.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But it wouldn't be a problem since OperationHelpers is internal, right?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How would you use it from other libraries if it stays internal?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point.

kinelski
kinelski commented Feb 23, 2021
View reviewed changes
sdk/core/Azure.Core.TestFramework/src/RecordedTestBase.cs
Comment on lines +137 to +138
[OneTimeSetUp]
public void UpdateDefaultPollingTime()
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there any point in restoring the original Polling Time as part of the tear down?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Another question: I don't think we currently have any test scenarios affected by that, but should we be concerned about adding a test flag to skip this step? This can be hard to accomplish since we'll have tests running in parallel.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not very comfortable with this approach. As you've said tests can be running in parallel and changing this setting might affect non-playback tests.

We have a common pattern where a single test is changed to record using a ctor parameter. If in that situation someone accidentally runs recorded tests along with the ones running live they'll flood the service with requests.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe it's OK to do on CI where we know the mode is global.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if we go a different way and instrument operations returned by clients the same way we instrument clients themselves. This way we can intercept the WaitForComplion call and pass 0 timeout to the base implementation.

@kinelski
Copy link
Member Author

/azp run net - core - ci

@azure-pipelines
Copy link

Azure Pipelines successfully started running 1 pipeline(s).

@check-enforcer
Copy link

This pull request is protected by Check Enforcer.

What is Check Enforcer?

Check Enforcer helps ensure all pull requests are covered by at least one check-run (typically an Azure Pipeline). When all check-runs associated with this pull request pass then Check Enforcer itself will pass.

Why am I getting this message?

You are getting this message because Check Enforcer did not detect any check-runs being associated with this pull request within five minutes. This may indicate that your pull request is not covered by any pipelines and so Check Enforcer is correctly blocking the pull request being merged.

What should I do now?

If the check-enforcer check-run is not passing and all other check-runs associated with this PR are passing (excluding license-cla) then you could try telling Check Enforcer to evaluate your pull request again. You can do this by adding a comment to this pull request as follows:
/check-enforcer evaluate
Typically evaulation only takes a few seconds. If you know that your pull request is not covered by a pipeline and this is expected you can override Check Enforcer using the following command:
/check-enforcer override
Note that using the override command triggers alerts so that follow-up investigations can occur (PRs still need to be approved as normal).

What if I am onboarding a new service?

Often, new services do not have validation pipelines associated with them. In order to bootstrap pipelines for a new service, please perform following steps:

For data-plane/track 2 SDKs Issue the following command as a pull request comment:

/azp run prepare-pipelines
This will run a pipeline that analyzes the source tree and creates the pipelines necessary to build and validate your pull request. Once the pipeline has been created you can trigger the pipeline using the following comment:
/azp run net - [service] - ci

For track 1 management-plane SDKs

Please open a separate PR and to your service SDK path in this file. Once that PR has been merged, you can re-run the pipeline to trigger the verification.

@kinelski kinelski requested review from pakrym and maririos February 23, 2021 20:57
@kinelski kinelski marked this pull request as ready for review February 23, 2021 20:57
@kinelski kinelski closed this Apr 6, 2021
@kinelski kinelski deleted the eng-poll branch April 6, 2021 17:51
azure-sdk pushed a commit to azure-sdk/azure-sdk-for-net that referenced this pull request May 24, 2022
@abhahn @bidisha-c
@ChongTang @moabba @nourhaanfarid
CodeGen from PR 19186 in Azure/azure-rest-api-specs
cbcd4c6 
Merge Cognitive Services - Language pillar version 2022-05-15-preview to main (Azure#19186)

* Updated May-1 preview to include changes from March-1 preview (Azure#18580)

* Updated May-1 preview to include changes from March-1 preview

* Fixed a missed api version update in an example file

Co-authored-by: Abigail Hartman <[email protected]>

* [Language API 2022-05-01-Preview]Conversation API Async - Conversational Summarization and Conversational PII (Azure#17847)

* Adds base for updating Language from version preview/2022-02-01-preview to version 2022-04-01-preview

* Updates readme

* Updates API version in new specs and examples

* [Language API 2022-04-01-Preview] Feature conversation PII redaction and issue resolution summarization

* Add Issue Resolution Summary

* Address comments

* address pretty chek and spell check issues

* address file name mismatch issue

* typo

* Add IRD updates

* Add IRD Schema changes

* Response Files content update

* Address feedback

* Address PR comments

* Address comments

* Add the May API version

* Address comments

* Remove 04-01 and address comments

* Update specification/cognitiveservices/data-plane/Language/preview/2022-05-01-preview/common.json

Co-authored-by: Heath Stewart <[email protected]>

* Update specification/cognitiveservices/data-plane/Language/preview/2022-05-01-preview/common.json

Co-authored-by: Heath Stewart <[email protected]>

* Update specification/cognitiveservices/data-plane/Language/preview/2022-05-01-preview/analyzeconversations.json

Co-authored-by: Heath Stewart <[email protected]>

* Update specification/cognitiveservices/data-plane/Language/preview/2022-05-01-preview/analyzeconversations.json

Co-authored-by: Heath Stewart <[email protected]>

* Update specification/cognitiveservices/data-plane/Language/preview/2022-05-01-preview/textanalytics.json

Co-authored-by: Heath Stewart <[email protected]>

* Address feedback V-1.

* Address all comments expect model + enum descriptions

* Fx validations

* Missed files

* Conversation Item modality issue

* Update PII Task

* Address Spec validation errors

* Updates

* Fix PII issues

* Add readonly

* Update schema

* Updates

* Bring back Text Conversation Item

* Update example

* Space in Json

* Address comments

* Address one more comments

* Fix model validation

* Fix

* Update spec

* Address comments

* Move requestStatistics, prebuiltResult, error and warnings object to common

* Address circular dependency

Co-authored-by: Abigail Hartman <[email protected]>
Co-authored-by: Heath Stewart <[email protected]>

* Rename version from 2022-05-01-preview to 2022-05-15-preview (Azure#18737)

* Update CLU swagger (Azure#18739)

Co-authored-by: Chong Tang <[email protected]>

* Use azure-sdk-for-net-track2 for validation (Azure#18755)

* Bidishac/update field description (#18787)

* Update the messaging on the parameter type

* Update the documentation of the redactionSource field

* Update specification/cognitiveservices/data-plane/Language/preview/2022-05-15-preview/analyzeconversations.json

Co-authored-by: Heath Stewart <[email protected]>

Co-authored-by: Chong Tang <[email protected]>
Co-authored-by: Heath Stewart <[email protected]>

* Unify the runtime and authoring APIs (Azure#18837)

* unifying single classification and multi classification to return array of class (Azure#18902)

* Renaming in 2022-05-15-preview (Azure#18929)

Co-authored-by: Chong Tang <[email protected]>

* Rename 'summary' in summaryAspects to 'issue' (Azure#18941)

* Rename 'summary' in summaryAspects to 'issue'

* Merge issues

* Update model version value

* Merge 2022-05-01-GA authoring updates and "LUIS" string case update into 2022-05-15-preview branch (Azure#18943)

* Merge 2022-05-01-GA files changes into 2022-05-15-preview branch

* Address Avocado issues and revert case update for Luis

* Add the missing error codes

* Fix avocado

* Avocado fix

* Add api-version parameter to cancel job endpoint (Azure#18986)

* Add api-version parameter to cancel job endpoint

* Fix taskState and jobState

* Revert "Fix taskState and jobState"

This reverts commit 148dd2183249add89a090721db36a14c2b15d9bd.

* Update LUIS to Luis in model names (Azure#19012)

* Copy QuestionAnswering definitions to common.json and refer them in analyzeconversations.json (Azure#19031)

Co-authored-by: Chong Tang <[email protected]>

* Added fhir feature updates to 2022-05-15-preview (Azure#19092)

Co-authored-by: Abigail Hartman <[email protected]>

* Add cancel endpoint for conversations (Azure#19025)

* Add cancel endpoint for conversations

* Update examples path

* Make the input for transcriptConversationItem optional

* Bidishac/update enum name (Azure#19149)

* Update name to CreditCard

* Update entity categories for PII for build

* Keep SSN

* Revert "Keep SSN"

This reverts commit 77a92119ebd71cf5d5c7f201d402514756289ec2.

* Update readme.md

Co-authored-by: Abigail Hartman <[email protected]>
Co-authored-by: Abigail Hartman <[email protected]>
Co-authored-by: Bidisha Chakraborty <[email protected]>
Co-authored-by: Chong Tang <[email protected]>
Co-authored-by: Chong Tang <[email protected]>
Co-authored-by: Mohamed Abbas <[email protected]>
Co-authored-by: Nourhan <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@pakrym pakrym Awaiting requested review from pakrym

@maririos maririos Awaiting requested review from maririos

Assignees

@kinelski kinelski

Labels
Azure.Core Client This issue points to a problem in the data-plane of the library. Cognitive - Form Recognizer Do Not Merge
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Set default polling time to zero when running on Playback mode
2 participants
@kinelski @pakrym