Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[TA] Fixes live test regression #24184

Merged
merged 3 commits into from
Sep 16, 2021

Conversation

mssfang
Copy link
Member

@mssfang mssfang commented Sep 16, 2021

There is a model change in Sept release as service team have moved to inference library and ORT 1.8. Regression is expected.

(1) For a specific test input: "Bonjour tout le monde." with language hint = "FR", The expected result for keyPhraseExtraction used to be "monde" but returns "Bonjour" now.

(2) For a list of input:Bonjour tout le monde. // only "monde" is returned when languageCode=FR, now the response has additional key phrase, "Bonjour" which is more reasonal. andJe m'appelle Mondly. // used to return nothing but now returns "Mondly"

(3) for a input string, Microsoft employee with ssn 859-98-0987 is using our awesome API's.The return key phrase list used to be "Microsoft employee", "ssn", "awesome", "API"But now it is "Microsoft employee", "ssn", "awesome API"

Create an issue to revisit defaultLanguage() method: #24183

@mssfang mssfang added Client This issue points to a problem in the data-plane of the library. Cognitive - Text Analytics labels Sep 16, 2021
@mssfang mssfang added this to the [2021] October milestone Sep 16, 2021
@mssfang mssfang self-assigned this Sep 16, 2021
@mssfang
Copy link
Member Author

mssfang commented Sep 16, 2021

/azp run java - textanalytics - tests

@azure-pipelines
Copy link

Azure Pipelines successfully started running 1 pipeline(s).

Copy link
Member

@maririos maririos left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Curious, is it common in Java libraries to validate the content of the output (especially a string) from the service?
In .NET we've had conversations about drawing a line between when we are testing the service per se and when we are actually testing the library.

@mssfang
Copy link
Member Author

mssfang commented Sep 16, 2021

Curious, is it common in Java libraries to validate the content of the output (especially a string) from the service?
In .NET we've had conversations about drawing a line between when we are testing the service per se and when we are actually testing the library.

Good point, it depends, if in AppConfig, if a configuration setting (k, v) sent to service, we need to validate if the response setting match with input. But in text analytics, there might not be always the case.

Create an issue to revisit the test strategy: #24186

@mssfang mssfang merged commit 2d76680 into Azure:main Sep 16, 2021
@mssfang mssfang deleted the TA-Fixed-live-test-regression branch September 16, 2021 22:42
srnagar pushed a commit to srnagar/azure-sdk-for-java that referenced this pull request Sep 20, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Client This issue points to a problem in the data-plane of the library. Cognitive - Text Analytics
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants