-
Notifications
You must be signed in to change notification settings - Fork 2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[TA] Fixes live test regression #24184
Conversation
…to TA-Fixed-live-test-regression
/azp run java - textanalytics - tests |
Azure Pipelines successfully started running 1 pipeline(s). |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Curious, is it common in Java libraries to validate the content of the output (especially a string) from the service?
In .NET we've had conversations about drawing a line between when we are testing the service per se and when we are actually testing the library.
Good point, it depends, if in AppConfig, if a configuration setting (k, v) sent to service, we need to validate if the response setting match with input. But in text analytics, there might not be always the case. Create an issue to revisit the test strategy: #24186 |
* fixed live test
There is a model change in Sept release as service team have moved to inference library and ORT 1.8. Regression is expected.
(1) For a specific test input: "Bonjour tout le monde." with language hint = "FR", The expected result for keyPhraseExtraction used to be "monde" but returns "Bonjour" now.
(2) For a list of input:Bonjour tout le monde. // only "monde" is returned when languageCode=FR, now the response has additional key phrase, "Bonjour" which is more reasonal. andJe m'appelle Mondly. // used to return nothing but now returns "Mondly"
(3) for a input string, Microsoft employee with ssn 859-98-0987 is using our awesome API's.The return key phrase list used to be "Microsoft employee", "ssn", "awesome", "API"But now it is "Microsoft employee", "ssn", "awesome API"
Create an issue to revisit defaultLanguage() method: #24183