From 7f062c4cb5f6b0393a6ef38142c76cbec18ebf99 Mon Sep 17 00:00:00 2001 From: Lalleh Rafeei <84813886+lrafeei@users.noreply.github.com> Date: Wed, 10 Jan 2024 15:17:14 -0800 Subject: [PATCH] Add bedrock feedback into preview (#1030) * Add AWS Bedrock testing infrastructure * Squashed commit of the following: commit 2834663794c649124052e510c1c9557a830c060a Author: Timothy Pansino <11214426+TimPansino@users.noreply.github.com> Date: Mon Oct 9 17:42:05 2023 -0700 OpenAI Mock Backend (#929) * Add mock external openai server * Add mocked OpenAI server fixtures * Set up recorded responses. * Clean mock server to depend on http server * Linting * Pin flask version for flask restx tests. (#931) * Ignore new redis methods. (#932) Co-authored-by: Lalleh Rafeei <84813886+lrafeei@users.noreply.github.com> * Remove approved paths * Update CI Image (#930) * Update available python versions in CI * Update makefile with overrides * Fix default branch detection for arm builds --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Add mocking for embedding endpoint * [Mega-Linter] Apply linters fixes * Add ratelimit headers * [Mega-Linter] Apply linters fixes * Only get package version once (#928) * Only get package version once * Add disconnect method * Add disconnect method --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Add datalib dependency for embedding testing. * Add OpenAI Test Infrastructure (#926) * Add openai to tox * Add OpenAI test files. * Add test functions. * [Mega-Linter] Apply linters fixes --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: mergify[bot] * Add mock external openai server * Add mocked OpenAI server fixtures * Set up recorded responses. * Clean mock server to depend on http server * Linting * Remove approved paths * Add mocking for embedding endpoint * [Mega-Linter] Apply linters fixes * Add ratelimit headers * [Mega-Linter] Apply linters fixes * Add datalib dependency for embedding testing. --------- Co-authored-by: Uma Annamalai Co-authored-by: Lalleh Rafeei <84813886+lrafeei@users.noreply.github.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: TimPansino Co-authored-by: Hannah Stepanek Co-authored-by: mergify[bot] commit db63d4598c94048986c0e00ebb2cd8827100b54c Author: Uma Annamalai Date: Mon Oct 2 15:31:38 2023 -0700 Add OpenAI Test Infrastructure (#926) * Add openai to tox * Add OpenAI test files. * Add test functions. * [Mega-Linter] Apply linters fixes --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: mergify[bot] * Squashed commit of the following: commit 182c7a8c8a91e2d0f234f7ed7d4a14a2422c8342 Author: Uma Annamalai Date: Fri Oct 13 10:12:55 2023 -0700 Add request/ response IDs. commit f6d13f822c22d2039ec32be86b2c54f9dc3de1c9 Author: Uma Annamalai Date: Thu Oct 12 13:23:39 2023 -0700 Test cleanup. commit d0576631d009e481bd5887a3243aac99b097d823 Author: Uma Annamalai Date: Tue Oct 10 10:23:00 2023 -0700 Remove commented code. commit dd29433e719482babbe5c724e7330b1f6324abd7 Author: Uma Annamalai Date: Tue Oct 10 10:19:01 2023 -0700 Add openai sync instrumentation. commit 2834663794c649124052e510c1c9557a830c060a Author: Timothy Pansino <11214426+TimPansino@users.noreply.github.com> Date: Mon Oct 9 17:42:05 2023 -0700 OpenAI Mock Backend (#929) * Add mock external openai server * Add mocked OpenAI server fixtures * Set up recorded responses. * Clean mock server to depend on http server * Linting * Pin flask version for flask restx tests. (#931) * Ignore new redis methods. (#932) Co-authored-by: Lalleh Rafeei <84813886+lrafeei@users.noreply.github.com> * Remove approved paths * Update CI Image (#930) * Update available python versions in CI * Update makefile with overrides * Fix default branch detection for arm builds --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Add mocking for embedding endpoint * [Mega-Linter] Apply linters fixes * Add ratelimit headers * [Mega-Linter] Apply linters fixes * Only get package version once (#928) * Only get package version once * Add disconnect method * Add disconnect method --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Add datalib dependency for embedding testing. * Add OpenAI Test Infrastructure (#926) * Add openai to tox * Add OpenAI test files. * Add test functions. * [Mega-Linter] Apply linters fixes --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: mergify[bot] * Add mock external openai server * Add mocked OpenAI server fixtures * Set up recorded responses. * Clean mock server to depend on http server * Linting * Remove approved paths * Add mocking for embedding endpoint * [Mega-Linter] Apply linters fixes * Add ratelimit headers * [Mega-Linter] Apply linters fixes * Add datalib dependency for embedding testing. --------- Co-authored-by: Uma Annamalai Co-authored-by: Lalleh Rafeei <84813886+lrafeei@users.noreply.github.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: TimPansino Co-authored-by: Hannah Stepanek Co-authored-by: mergify[bot] commit db63d4598c94048986c0e00ebb2cd8827100b54c Author: Uma Annamalai Date: Mon Oct 2 15:31:38 2023 -0700 Add OpenAI Test Infrastructure (#926) * Add openai to tox * Add OpenAI test files. * Add test functions. * [Mega-Linter] Apply linters fixes --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: mergify[bot] * TEMP * Bedrock titan extraction nearly complete * Bedrock Testing Infrastructure (#937) * Add AWS Bedrock testing infrastructure * Cache Package Version Lookups (#946) * Cache _get_package_version * Add Python 2.7 support to get_package_version caching * [Mega-Linter] Apply linters fixes * Bump tests --------- Co-authored-by: SlavaSkvortsov <29122694+SlavaSkvortsov@users.noreply.github.com> Co-authored-by: TimPansino * Fix Redis Generator Methods (#947) * Fix scan_iter for redis * Replace generator methods * Update instance info instrumentation * Remove mistake from uninstrumented methods * Add skip condition to asyncio generator tests * Add skip condition to asyncio generator tests --------- Co-authored-by: Lalleh Rafeei Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Automatic RPM System Updates (#948) * Checkout old action * Adding RPM action * Add dry run * Incorporating action into workflow * Wire secret into custom action * Enable action * Correct action name * Fix syntax * Fix quoting issues * Drop pre-verification. Does not work on python * Fix merge artifact * Remove OpenAI references --------- Co-authored-by: Uma Annamalai Co-authored-by: SlavaSkvortsov <29122694+SlavaSkvortsov@users.noreply.github.com> Co-authored-by: TimPansino Co-authored-by: Lalleh Rafeei Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Cleaning up titan bedrock implementation * TEMP * Tests for bedrock passing Co-authored-by: Lalleh Rafeei * Cleaned up titan testing Co-authored-by: Lalleh Rafeei Co-authored-by: Hannah Stepanek * Parametrized bedrock testing * Add support for AI21-J2 models * Change to dynamic no conversation id events * Drop all openai refs * [Mega-Linter] Apply linters fixes * Adding response_id and response_model * Apply suggestions from code review * Remove unused import * Bedrock Sync Chat Completion Instrumentation (#953) * Add AWS Bedrock testing infrastructure * Squashed commit of the following: commit 2834663794c649124052e510c1c9557a830c060a Author: Timothy Pansino <11214426+TimPansino@users.noreply.github.com> Date: Mon Oct 9 17:42:05 2023 -0700 OpenAI Mock Backend (#929) * Add mock external openai server * Add mocked OpenAI server fixtures * Set up recorded responses. * Clean mock server to depend on http server * Linting * Pin flask version for flask restx tests. (#931) * Ignore new redis methods. (#932) Co-authored-by: Lalleh Rafeei <84813886+lrafeei@users.noreply.github.com> * Remove approved paths * Update CI Image (#930) * Update available python versions in CI * Update makefile with overrides * Fix default branch detection for arm builds --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Add mocking for embedding endpoint * [Mega-Linter] Apply linters fixes * Add ratelimit headers * [Mega-Linter] Apply linters fixes * Only get package version once (#928) * Only get package version once * Add disconnect method * Add disconnect method --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Add datalib dependency for embedding testing. * Add OpenAI Test Infrastructure (#926) * Add openai to tox * Add OpenAI test files. * Add test functions. * [Mega-Linter] Apply linters fixes --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: mergify[bot] * Add mock external openai server * Add mocked OpenAI server fixtures * Set up recorded responses. * Clean mock server to depend on http server * Linting * Remove approved paths * Add mocking for embedding endpoint * [Mega-Linter] Apply linters fixes * Add ratelimit headers * [Mega-Linter] Apply linters fixes * Add datalib dependency for embedding testing. --------- Co-authored-by: Uma Annamalai Co-authored-by: Lalleh Rafeei <84813886+lrafeei@users.noreply.github.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: TimPansino Co-authored-by: Hannah Stepanek Co-authored-by: mergify[bot] commit db63d4598c94048986c0e00ebb2cd8827100b54c Author: Uma Annamalai Date: Mon Oct 2 15:31:38 2023 -0700 Add OpenAI Test Infrastructure (#926) * Add openai to tox * Add OpenAI test files. * Add test functions. * [Mega-Linter] Apply linters fixes --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: mergify[bot] * Squashed commit of the following: commit 182c7a8c8a91e2d0f234f7ed7d4a14a2422c8342 Author: Uma Annamalai Date: Fri Oct 13 10:12:55 2023 -0700 Add request/ response IDs. commit f6d13f822c22d2039ec32be86b2c54f9dc3de1c9 Author: Uma Annamalai Date: Thu Oct 12 13:23:39 2023 -0700 Test cleanup. commit d0576631d009e481bd5887a3243aac99b097d823 Author: Uma Annamalai Date: Tue Oct 10 10:23:00 2023 -0700 Remove commented code. commit dd29433e719482babbe5c724e7330b1f6324abd7 Author: Uma Annamalai Date: Tue Oct 10 10:19:01 2023 -0700 Add openai sync instrumentation. commit 2834663794c649124052e510c1c9557a830c060a Author: Timothy Pansino <11214426+TimPansino@users.noreply.github.com> Date: Mon Oct 9 17:42:05 2023 -0700 OpenAI Mock Backend (#929) * Add mock external openai server * Add mocked OpenAI server fixtures * Set up recorded responses. * Clean mock server to depend on http server * Linting * Pin flask version for flask restx tests. (#931) * Ignore new redis methods. (#932) Co-authored-by: Lalleh Rafeei <84813886+lrafeei@users.noreply.github.com> * Remove approved paths * Update CI Image (#930) * Update available python versions in CI * Update makefile with overrides * Fix default branch detection for arm builds --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Add mocking for embedding endpoint * [Mega-Linter] Apply linters fixes * Add ratelimit headers * [Mega-Linter] Apply linters fixes * Only get package version once (#928) * Only get package version once * Add disconnect method * Add disconnect method --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Add datalib dependency for embedding testing. * Add OpenAI Test Infrastructure (#926) * Add openai to tox * Add OpenAI test files. * Add test functions. * [Mega-Linter] Apply linters fixes --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: mergify[bot] * Add mock external openai server * Add mocked OpenAI server fixtures * Set up recorded responses. * Clean mock server to depend on http server * Linting * Remove approved paths * Add mocking for embedding endpoint * [Mega-Linter] Apply linters fixes * Add ratelimit headers * [Mega-Linter] Apply linters fixes * Add datalib dependency for embedding testing. --------- Co-authored-by: Uma Annamalai Co-authored-by: Lalleh Rafeei <84813886+lrafeei@users.noreply.github.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: TimPansino Co-authored-by: Hannah Stepanek Co-authored-by: mergify[bot] commit db63d4598c94048986c0e00ebb2cd8827100b54c Author: Uma Annamalai Date: Mon Oct 2 15:31:38 2023 -0700 Add OpenAI Test Infrastructure (#926) * Add openai to tox * Add OpenAI test files. * Add test functions. * [Mega-Linter] Apply linters fixes --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: mergify[bot] * Cache Package Version Lookups (#946) * Cache _get_package_version * Add Python 2.7 support to get_package_version caching * [Mega-Linter] Apply linters fixes * Bump tests --------- Co-authored-by: SlavaSkvortsov <29122694+SlavaSkvortsov@users.noreply.github.com> Co-authored-by: TimPansino * Fix Redis Generator Methods (#947) * Fix scan_iter for redis * Replace generator methods * Update instance info instrumentation * Remove mistake from uninstrumented methods * Add skip condition to asyncio generator tests * Add skip condition to asyncio generator tests --------- Co-authored-by: Lalleh Rafeei Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * TEMP * Automatic RPM System Updates (#948) * Checkout old action * Adding RPM action * Add dry run * Incorporating action into workflow * Wire secret into custom action * Enable action * Correct action name * Fix syntax * Fix quoting issues * Drop pre-verification. Does not work on python * Fix merge artifact * Bedrock titan extraction nearly complete * Cleaning up titan bedrock implementation * TEMP * Tests for bedrock passing Co-authored-by: Lalleh Rafeei * Cleaned up titan testing Co-authored-by: Lalleh Rafeei Co-authored-by: Hannah Stepanek * Parametrized bedrock testing * Add support for AI21-J2 models * Change to dynamic no conversation id events * Drop all openai refs * [Mega-Linter] Apply linters fixes * Adding response_id and response_model * Drop python 3.7 tests for Hypercorn (#954) * Apply suggestions from code review * Remove unused import --------- Co-authored-by: Uma Annamalai Co-authored-by: SlavaSkvortsov <29122694+SlavaSkvortsov@users.noreply.github.com> Co-authored-by: TimPansino Co-authored-by: Lalleh Rafeei Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: Lalleh Rafeei Co-authored-by: Hannah Stepanek Co-authored-by: Lalleh Rafeei <84813886+lrafeei@users.noreply.github.com> * Initial feedback commit for botocore * Feature bedrock cohere instrumentation (#955) * Add AWS Bedrock testing infrastructure * Squashed commit of the following: commit 2834663794c649124052e510c1c9557a830c060a Author: Timothy Pansino <11214426+TimPansino@users.noreply.github.com> Date: Mon Oct 9 17:42:05 2023 -0700 OpenAI Mock Backend (#929) * Add mock external openai server * Add mocked OpenAI server fixtures * Set up recorded responses. * Clean mock server to depend on http server * Linting * Pin flask version for flask restx tests. (#931) * Ignore new redis methods. (#932) Co-authored-by: Lalleh Rafeei <84813886+lrafeei@users.noreply.github.com> * Remove approved paths * Update CI Image (#930) * Update available python versions in CI * Update makefile with overrides * Fix default branch detection for arm builds --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Add mocking for embedding endpoint * [Mega-Linter] Apply linters fixes * Add ratelimit headers * [Mega-Linter] Apply linters fixes * Only get package version once (#928) * Only get package version once * Add disconnect method * Add disconnect method --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Add datalib dependency for embedding testing. * Add OpenAI Test Infrastructure (#926) * Add openai to tox * Add OpenAI test files. * Add test functions. * [Mega-Linter] Apply linters fixes --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: mergify[bot] * Add mock external openai server * Add mocked OpenAI server fixtures * Set up recorded responses. * Clean mock server to depend on http server * Linting * Remove approved paths * Add mocking for embedding endpoint * [Mega-Linter] Apply linters fixes * Add ratelimit headers * [Mega-Linter] Apply linters fixes * Add datalib dependency for embedding testing. --------- Co-authored-by: Uma Annamalai Co-authored-by: Lalleh Rafeei <84813886+lrafeei@users.noreply.github.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: TimPansino Co-authored-by: Hannah Stepanek Co-authored-by: mergify[bot] commit db63d4598c94048986c0e00ebb2cd8827100b54c Author: Uma Annamalai Date: Mon Oct 2 15:31:38 2023 -0700 Add OpenAI Test Infrastructure (#926) * Add openai to tox * Add OpenAI test files. * Add test functions. * [Mega-Linter] Apply linters fixes --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: mergify[bot] * Squashed commit of the following: commit 182c7a8c8a91e2d0f234f7ed7d4a14a2422c8342 Author: Uma Annamalai Date: Fri Oct 13 10:12:55 2023 -0700 Add request/ response IDs. commit f6d13f822c22d2039ec32be86b2c54f9dc3de1c9 Author: Uma Annamalai Date: Thu Oct 12 13:23:39 2023 -0700 Test cleanup. commit d0576631d009e481bd5887a3243aac99b097d823 Author: Uma Annamalai Date: Tue Oct 10 10:23:00 2023 -0700 Remove commented code. commit dd29433e719482babbe5c724e7330b1f6324abd7 Author: Uma Annamalai Date: Tue Oct 10 10:19:01 2023 -0700 Add openai sync instrumentation. commit 2834663794c649124052e510c1c9557a830c060a Author: Timothy Pansino <11214426+TimPansino@users.noreply.github.com> Date: Mon Oct 9 17:42:05 2023 -0700 OpenAI Mock Backend (#929) * Add mock external openai server * Add mocked OpenAI server fixtures * Set up recorded responses. * Clean mock server to depend on http server * Linting * Pin flask version for flask restx tests. (#931) * Ignore new redis methods. (#932) Co-authored-by: Lalleh Rafeei <84813886+lrafeei@users.noreply.github.com> * Remove approved paths * Update CI Image (#930) * Update available python versions in CI * Update makefile with overrides * Fix default branch detection for arm builds --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Add mocking for embedding endpoint * [Mega-Linter] Apply linters fixes * Add ratelimit headers * [Mega-Linter] Apply linters fixes * Only get package version once (#928) * Only get package version once * Add disconnect method * Add disconnect method --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Add datalib dependency for embedding testing. * Add OpenAI Test Infrastructure (#926) * Add openai to tox * Add OpenAI test files. * Add test functions. * [Mega-Linter] Apply linters fixes --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: mergify[bot] * Add mock external openai server * Add mocked OpenAI server fixtures * Set up recorded responses. * Clean mock server to depend on http server * Linting * Remove approved paths * Add mocking for embedding endpoint * [Mega-Linter] Apply linters fixes * Add ratelimit headers * [Mega-Linter] Apply linters fixes * Add datalib dependency for embedding testing. --------- Co-authored-by: Uma Annamalai Co-authored-by: Lalleh Rafeei <84813886+lrafeei@users.noreply.github.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: TimPansino Co-authored-by: Hannah Stepanek Co-authored-by: mergify[bot] commit db63d4598c94048986c0e00ebb2cd8827100b54c Author: Uma Annamalai Date: Mon Oct 2 15:31:38 2023 -0700 Add OpenAI Test Infrastructure (#926) * Add openai to tox * Add OpenAI test files. * Add test functions. * [Mega-Linter] Apply linters fixes --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: mergify[bot] * TEMP * Bedrock titan extraction nearly complete * Cleaning up titan bedrock implementation * TEMP * Tests for bedrock passing Co-authored-by: Lalleh Rafeei * Cleaned up titan testing Co-authored-by: Lalleh Rafeei Co-authored-by: Hannah Stepanek * Parametrized bedrock testing * Add support for AI21-J2 models * Change to dynamic no conversation id events * Add cohere model * Remove openai instrumentation from this branch * Remove OpenAI from newrelic/config.py --------- Co-authored-by: Uma Annamalai Co-authored-by: Tim Pansino Co-authored-by: Lalleh Rafeei Co-authored-by: Hannah Stepanek * Bedrock feedback w/ testing for titan and jurassic models * AWS Bedrock Embedding Instrumentation (#957) * AWS Bedrock embedding instrumentation * Correct symbol name * Add support for bedrock claude (#960) Co-authored-by: Timothy Pansino <11214426+TimPansino@users.noreply.github.com> * Fix merge conflicts * Combine Botocore Tests (#959) * Initial file migration * Enable DT on all span tests * Add pytest skip for older botocore versions * Fixup: app name merge conflict --------- Co-authored-by: Hannah Stepanek * Add to and move feedback tests * Handle 0.32.0.post1 version in tests (#963) * Remove response_id dependency in bedrock * Change API name * Update moto * Bedrock Error Tracing (#966) * Cache Package Version Lookups (#946) * Cache _get_package_version * Add Python 2.7 support to get_package_version caching * [Mega-Linter] Apply linters fixes * Bump tests --------- Co-authored-by: SlavaSkvortsov <29122694+SlavaSkvortsov@users.noreply.github.com> Co-authored-by: TimPansino * Fix Redis Generator Methods (#947) * Fix scan_iter for redis * Replace generator methods * Update instance info instrumentation * Remove mistake from uninstrumented methods * Add skip condition to asyncio generator tests * Add skip condition to asyncio generator tests --------- Co-authored-by: Lalleh Rafeei Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Automatic RPM System Updates (#948) * Checkout old action * Adding RPM action * Add dry run * Incorporating action into workflow * Wire secret into custom action * Enable action * Correct action name * Fix syntax * Fix quoting issues * Drop pre-verification. Does not work on python * Fix merge artifact * Drop python 3.7 tests for Hypercorn (#954) * Fix pyenv installation for devcontainer (#936) Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Remove duplicate kafka import hook (#956) Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Initial bedrock error tracing commit * Handle 0.32.0.post1 version in tests (#963) * Add status code to mock bedrock server * Updating error response recording logic * Work on bedrock errror tracing * Chat completion error tracing * Adding embedding error tracing * Delete comment * Update moto --------- Co-authored-by: SlavaSkvortsov <29122694+SlavaSkvortsov@users.noreply.github.com> Co-authored-by: TimPansino Co-authored-by: Lalleh Rafeei Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: Lalleh Rafeei <84813886+lrafeei@users.noreply.github.com> Co-authored-by: Hannah Stepanek * Change ids to match other tests * move message_ids declaration outside for loop * Add comment to tox.ini * Drop py27 from memcache testing. * Drop pypy27 from memcache testing. * Update flaskrestx testing #1004 * Remove tastypie 0.14.3 testing * Remove tastypie 0.14.3 testing * Remove python 3.12 support (for now) * Remove untouched files from diff list --------- Co-authored-by: Uma Annamalai Co-authored-by: Tim Pansino Co-authored-by: Timothy Pansino <11214426+TimPansino@users.noreply.github.com> Co-authored-by: SlavaSkvortsov <29122694+SlavaSkvortsov@users.noreply.github.com> Co-authored-by: TimPansino Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: Lalleh Rafeei Co-authored-by: Hannah Stepanek Co-authored-by: Hannah Stepanek --- newrelic/api/ml_model.py | 9 +- newrelic/hooks/external_botocore.py | 9 +- .../_test_bedrock_chat_completion.py | 87 +++++++++++++++++++ .../_test_bedrock_embeddings.py | 18 +++- .../test_bedrock_embeddings.py | 2 +- tox.ini | 23 +++-- 6 files changed, 129 insertions(+), 19 deletions(-) diff --git a/newrelic/api/ml_model.py b/newrelic/api/ml_model.py index 3d15cf8d37..03408253bc 100644 --- a/newrelic/api/ml_model.py +++ b/newrelic/api/ml_model.py @@ -40,12 +40,15 @@ def wrap_mlmodel(model, name=None, version=None, feature_names=None, label_names def get_llm_message_ids(response_id=None): transaction = current_transaction() - if response_id and transaction: + if transaction: nr_message_ids = getattr(transaction, "_nr_message_ids", {}) - message_id_info = nr_message_ids.pop(response_id, ()) + message_id_info = ( + nr_message_ids.pop("bedrock_key", ()) if not response_id else nr_message_ids.pop(response_id, ()) + ) if not message_id_info: - warnings.warn("No message ids found for %s" % response_id) + response_id_warning = "." if not response_id else " for %s." % response_id + warnings.warn("No message ids found%s" % response_id_warning) return [] conversation_id, request_id, ids = message_id_info diff --git a/newrelic/hooks/external_botocore.py b/newrelic/hooks/external_botocore.py index 6e3be661bd..69a2fd9361 100644 --- a/newrelic/hooks/external_botocore.py +++ b/newrelic/hooks/external_botocore.py @@ -97,6 +97,7 @@ def create_chat_completion_message_event( if not transaction: return + message_ids = [] for index, message in enumerate(input_message_list): if response_id: id_ = "%s-%d" % (response_id, index) # Response ID was set, append message index to it. @@ -128,6 +129,7 @@ def create_chat_completion_message_event( id_ = "%s-%d" % (response_id, index) # Response ID was set, append message index to it. else: id_ = str(uuid.uuid4()) # No response IDs, use random UUID + message_ids.append(id_) chat_completion_message_dict = { "id": id_, @@ -147,6 +149,7 @@ def create_chat_completion_message_event( "is_response": True, } transaction.record_custom_event("LlmChatCompletionMessage", chat_completion_message_dict) + return (conversation_id, request_id, message_ids) def extract_bedrock_titan_text_model(request_body, response_body=None): @@ -577,7 +580,7 @@ def handle_chat_completion_event( transaction.record_custom_event("LlmChatCompletionSummary", chat_completion_summary_dict) - create_chat_completion_message_event( + message_ids = create_chat_completion_message_event( transaction=transaction, app_name=settings.app_name, input_message_list=input_message_list, @@ -591,6 +594,10 @@ def handle_chat_completion_event( response_id=response_id, ) + if not hasattr(transaction, "_nr_message_ids"): + transaction._nr_message_ids = {} + transaction._nr_message_ids["bedrock_key"] = message_ids + CUSTOM_TRACE_POINTS = { ("sns", "publish"): message_trace("SNS", "Produce", "Topic", extract(("TopicArn", "TargetArn"), "PhoneNumber")), diff --git a/tests/external_botocore/_test_bedrock_chat_completion.py b/tests/external_botocore/_test_bedrock_chat_completion.py index f1d21c73c7..652027719c 100644 --- a/tests/external_botocore/_test_bedrock_chat_completion.py +++ b/tests/external_botocore/_test_bedrock_chat_completion.py @@ -1,3 +1,17 @@ +# Copyright 2010 New Relic, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + chat_completion_payload_templates = { "amazon.titan-text-express-v1": '{ "inputText": "%s", "textGenerationConfig": {"temperature": %f, "maxTokenCount": %d }}', "ai21.j2-mid-v1": '{"prompt": "%s", "temperature": %f, "maxTokens": %d}', @@ -6,6 +20,79 @@ "meta.llama2-13b-chat-v1": '{"prompt": "%s", "temperature": %f, "max_gen_len": %d}', } +chat_completion_get_llm_message_ids = { + "amazon.titan-text-express-v1": { + "bedrock_key": [ + { + "conversation_id": "my-awesome-id", + "request_id": "03524118-8d77-430f-9e08-63b5c03a40cf", + "message_id": None, # UUID that varies with each run + }, + { + "conversation_id": "my-awesome-id", + "request_id": "03524118-8d77-430f-9e08-63b5c03a40cf", + "message_id": None, # UUID that varies with each run + }, + ] + }, + "ai21.j2-mid-v1": { + "bedrock_key": [ + { + "conversation_id": "my-awesome-id", + "request_id": "c863d9fc-888b-421c-a175-ac5256baec62", + "message_id": "1234-0", + }, + { + "conversation_id": "my-awesome-id", + "request_id": "c863d9fc-888b-421c-a175-ac5256baec62", + "message_id": "1234-1", + }, + ] + }, + "anthropic.claude-instant-v1": { + "bedrock_key": [ + { + "conversation_id": "my-awesome-id", + "request_id": "7b0b37c6-85fb-4664-8f5b-361ca7b1aa18", + "message_id": None, # UUID that varies with each run + }, + { + "conversation_id": "my-awesome-id", + "request_id": "7b0b37c6-85fb-4664-8f5b-361ca7b1aa18", + "message_id": None, # UUID that varies with each run + }, + ] + }, + "cohere.command-text-v14": { + "bedrock_key": [ + { + "conversation_id": "my-awesome-id", + "request_id": "e77422c8-fbbf-4e17-afeb-c758425c9f97", + "message_id": "e77422c8-fbbf-4e17-afeb-c758425c9f97-0", + }, + { + "conversation_id": "my-awesome-id", + "request_id": "e77422c8-fbbf-4e17-afeb-c758425c9f97", + "message_id": "e77422c8-fbbf-4e17-afeb-c758425c9f97-1", + }, + ] + }, + "meta.llama2-13b-chat-v1": { + "bedrock_key": [ + { + "conversation_id": "my-awesome-id", + "request_id": "9a64cdb0-3e82-41c7-873a-c12a77e0143a", + "message_id": "9a64cdb0-3e82-41c7-873a-c12a77e0143a-0", + }, + { + "conversation_id": "my-awesome-id", + "request_id": "9a64cdb0-3e82-41c7-873a-c12a77e0143a", + "message_id": "9a64cdb0-3e82-41c7-873a-c12a77e0143a-1", + }, + ] + }, +} + chat_completion_expected_events = { "amazon.titan-text-express-v1": [ ( diff --git a/tests/external_botocore/_test_bedrock_embeddings.py b/tests/external_botocore/_test_bedrock_embeddings.py index ec677b426c..05c8a390ca 100644 --- a/tests/external_botocore/_test_bedrock_embeddings.py +++ b/tests/external_botocore/_test_bedrock_embeddings.py @@ -1,3 +1,17 @@ +# Copyright 2010 New Relic, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + embedding_payload_templates = { "amazon.titan-embed-text-v1": '{ "inputText": "%s" }', "amazon.titan-embed-g1-text-02": '{ "inputText": "%s" }', @@ -68,7 +82,7 @@ "request_id": "", "vendor": "bedrock", "ingest_source": "Python", - "error": True + "error": True, }, ), ], @@ -89,7 +103,7 @@ "request_id": "", "vendor": "bedrock", "ingest_source": "Python", - "error": True + "error": True, }, ), ], diff --git a/tests/external_botocore/test_bedrock_embeddings.py b/tests/external_botocore/test_bedrock_embeddings.py index cc442fc158..9fc0164714 100644 --- a/tests/external_botocore/test_bedrock_embeddings.py +++ b/tests/external_botocore/test_bedrock_embeddings.py @@ -19,8 +19,8 @@ import pytest from _test_bedrock_embeddings import ( embedding_expected_client_errors, - embedding_expected_events, embedding_expected_error_events, + embedding_expected_events, embedding_payload_templates, ) from conftest import BOTOCORE_VERSION diff --git a/tox.ini b/tox.ini index a0827dea61..bc54db2b92 100644 --- a/tox.ini +++ b/tox.ini @@ -67,11 +67,11 @@ envlist = python-mlmodel_sklearn-{py37}-scikitlearn0101, python-component_djangorestframework-py27-djangorestframework0300, python-component_djangorestframework-{py37,py38,py39,py310,py311}-djangorestframeworklatest, - python-component_flask_rest-{py37,py38,py39,pypy38}-flaskrestxlatest, + python-component_flask_rest-py37-flaskrestx110, + python-component_flask_rest-{py38,py39,py310,py311,pypy38}-flaskrestxlatest, python-component_flask_rest-{py27,pypy27}-flaskrestx051, python-component_graphqlserver-{py37,py38,py39,py310,py311}, - python-component_tastypie-{py27,pypy27}-tastypie0143, - python-component_tastypie-{py37,py38,py39,pypy38}-tastypie{0143,latest}, + python-component_tastypie-{py37,py38,py39,pypy38}-tastypielatest, python-coroutines_asyncio-{py37,py38,py39,py310,py311,pypy38}, python-cross_agent-{py27,py37,py38,py39,py310,py311}-{with,without}_extensions, python-cross_agent-pypy27-without_extensions, @@ -79,7 +79,7 @@ envlist = memcached-datastore_bmemcached-{pypy27,py27,py37,py38,py39,py310,py311}-memcached030, elasticsearchserver07-datastore_elasticsearch-{py27,py37,py38,py39,py310,py311,pypy27,pypy38}-elasticsearch07, elasticsearchserver08-datastore_elasticsearch-{py37,py38,py39,py310,py311,pypy38}-elasticsearch08, - memcached-datastore_memcache-{py27,py37,py38,py39,py310,py311,pypy27,pypy38}-memcached01, + memcached-datastore_memcache-{py37,py38,py39,py310,py311,pypy38}-memcached01, mysql-datastore_mysql-mysql080023-py27, mysql-datastore_mysql-mysqllatest-{py37,py38,py39,py310,py311}, firestore-datastore_firestore-{py37,py38,py39,py310,py311}, @@ -177,7 +177,8 @@ deps = adapter_gunicorn-aiohttp3: aiohttp<4.0 adapter_gunicorn-gunicorn19: gunicorn<20 adapter_gunicorn-gunicornlatest: gunicorn - adapter_hypercorn-hypercornlatest: hypercorn + ; Temporarily pinned. Needs to be addressed + adapter_hypercorn-hypercornlatest: hypercorn<0.16 adapter_hypercorn-hypercorn0013: hypercorn<0.14 adapter_hypercorn-hypercorn0012: hypercorn<0.13 adapter_hypercorn-hypercorn0011: hypercorn<0.12 @@ -204,23 +205,21 @@ deps = component_djangorestframework-djangorestframework0300: djangorestframework<3.1 component_djangorestframework-djangorestframeworklatest: Django component_djangorestframework-djangorestframeworklatest: djangorestframework - component_flask_rest: flask component_flask_rest: flask-restful component_flask_rest: jinja2 component_flask_rest: itsdangerous + component_flask_rest-flaskrestxlatest: flask component_flask_rest-flaskrestxlatest: flask-restx - ; Pin Flask version until flask-restx is updated to support v3 - component_flask_rest-flaskrestxlatest: flask<3.0 + ; flask-restx only supports Flask v3 after flask-restx v1.3.0 + component_flask_rest-flaskrestx110: Flask<3.0 + component_flask_rest-flaskrestx110: flask-restx<1.2 + component_flask_rest-flaskrestx051: Flask<3.0 component_flask_rest-flaskrestx051: flask-restx<1.0 component_graphqlserver: graphql-server[sanic,flask]==3.0.0b5 component_graphqlserver: sanic>20 component_graphqlserver: Flask component_graphqlserver: markupsafe<2.1 component_graphqlserver: jinja2<3.1 - component_tastypie-tastypie0143: django-tastypie<0.14.4 - component_tastypie-{py27,pypy27}-tastypie0143: django<1.12 - component_tastypie-{py37,py38,py39,py310,py311,pypy38}-tastypie0143: django<3.0.1 - component_tastypie-{py37,py38,py39,py310,py311,pypy38}-tastypie0143: asgiref<3.7.1 # asgiref==3.7.1 only suppport Python 3.10+ component_tastypie-tastypielatest: django-tastypie component_tastypie-tastypielatest: django<4.1 component_tastypie-tastypielatest: asgiref<3.7.1 # asgiref==3.7.1 only suppport Python 3.10+