Switch AI instrumentation to use custom events (#974)

* Add OpenAI Test Infrastructure (#926) * Add openai to tox * Add OpenAI test files. * Add test functions. * [Mega-Linter] Apply linters fixes --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: mergify[bot] <mergify[bot]@users.noreply.github.com> * OpenAI Mock Backend (#929) * Add mock external openai server * Add mocked OpenAI server fixtures * Set up recorded responses. * Clean mock server to depend on http server * Linting * Pin flask version for flask restx tests. (#931) * Ignore new redis methods. (#932) Co-authored-by: Lalleh Rafeei <[email protected]> * Remove approved paths * Update CI Image (#930) * Update available python versions in CI * Update makefile with overrides * Fix default branch detection for arm builds --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Add mocking for embedding endpoint * [Mega-Linter] Apply linters fixes * Add ratelimit headers * [Mega-Linter] Apply linters fixes * Only get package version once (#928) * Only get package version once * Add disconnect method * Add disconnect method --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Add datalib dependency for embedding testing. * Add OpenAI Test Infrastructure (#926) * Add openai to tox * Add OpenAI test files. * Add test functions. * [Mega-Linter] Apply linters fixes --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: mergify[bot] <mergify[bot]@users.noreply.github.com> * Add mock external openai server * Add mocked OpenAI server fixtures * Set up recorded responses. * Clean mock server to depend on http server * Linting * Remove approved paths * Add mocking for embedding endpoint * [Mega-Linter] Apply linters fixes * Add ratelimit headers * [Mega-Linter] Apply linters fixes * Add datalib dependency for embedding testing. --------- Co-authored-by: Uma Annamalai <[email protected]> Co-authored-by: Lalleh Rafeei <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: TimPansino <[email protected]> Co-authored-by: Hannah Stepanek <[email protected]> Co-authored-by: mergify[bot] <mergify[bot]@users.noreply.github.com> * Update OpenAI testing infra to match bedrock (#939) * Add OpenAI sync chat completion instrumentation (#934) * Add openai sync instrumentation. * Remove commented code. * Test cleanup. * Add request/ response IDs. * Fixups. * Add conversation ID to message events. --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Add OpenAI sync embedding instrumentation (#938) * Add sync instrumentation for OpenAI embeddings. * Remove comments. * Clean up embedding event dictionary. * Update response_time to duration. * Linting fixes. * [Mega-Linter] Apply linters fixes * Trigger tests --------- Co-authored-by: umaannamalai <[email protected]> Co-authored-by: Hannah Stepanek <[email protected]> * Instrument acreate's for open-ai (#935) * Instrument acreate's for open ai async * Remove duplicated vendor * Re-use expected & input payloads in tests * Attach ml_event to APM entity by default (#940) * Attach non InferenceEvents to APM entity * Validate both resource payloads * Add tests for non-inference events * Add OpenAI sync embedding instrumentation (#938) * Add sync instrumentation for OpenAI embeddings. * Remove comments. * Clean up embedding event dictionary. * Update response_time to duration. * Linting fixes. * [Mega-Linter] Apply linters fixes * Trigger tests --------- Co-authored-by: umaannamalai <[email protected]> Co-authored-by: Hannah Stepanek <[email protected]> * Fixup: test names --------- Co-authored-by: Uma Annamalai <[email protected]> Co-authored-by: umaannamalai <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Add truncation for ML events. (#943) * Add 4096 char truncation for ML events. * Add max attr check. * Fixup. * Fix character length ml event test. * Ignore test_ml_events.py for Py2. * Cleanup custom event if checks. * Add import statement. --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Add framework metric for OpenAI. (#945) * Add framework metric for OpenAI. * [Mega-Linter] Apply linters fixes * Trigger tests * Fix missing version info. * [Mega-Linter] Apply linters fixes --------- Co-authored-by: umaannamalai <[email protected]> Co-authored-by: Hannah Stepanek <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Add truncation support for ML events recorded outside txns. (#949) * Add ml tests for outside transaction. * Update validator. * Add ML flag to application code path for record_ml_event. * Bedrock Testing Infrastructure (#937) * Add AWS Bedrock testing infrastructure * Cache Package Version Lookups (#946) * Cache _get_package_version * Add Python 2.7 support to get_package_version caching * [Mega-Linter] Apply linters fixes * Bump tests --------- Co-authored-by: SlavaSkvortsov <[email protected]> Co-authored-by: TimPansino <[email protected]> * Fix Redis Generator Methods (#947) * Fix scan_iter for redis * Replace generator methods * Update instance info instrumentation * Remove mistake from uninstrumented methods * Add skip condition to asyncio generator tests * Add skip condition to asyncio generator tests --------- Co-authored-by: Lalleh Rafeei <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Automatic RPM System Updates (#948) * Checkout old action * Adding RPM action * Add dry run * Incorporating action into workflow * Wire secret into custom action * Enable action * Correct action name * Fix syntax * Fix quoting issues * Drop pre-verification. Does not work on python * Fix merge artifact * Remove OpenAI references --------- Co-authored-by: Uma Annamalai <[email protected]> Co-authored-by: SlavaSkvortsov <[email protected]> Co-authored-by: TimPansino <[email protected]> Co-authored-by: Lalleh Rafeei <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Mock openai error responses (#950) * Add example tests and mock error responses * Set invalid api key in auth error test Co-authored-by: Timothy Pansino <[email protected]> --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: Timothy Pansino <[email protected]> * OpenAI ErrorTrace attributes (#941) * Add openai sync instrumentation. * Remove commented code. * Initial openai error commit * Add example tests and mock error responses * Changes to attribute collection * Change error tests to match mock server * [Mega-Linter] Apply linters fixes * Trigger tests * Add dt_enabled decorator to error tests * Add embedded and async error tests * [Mega-Linter] Apply linters fixes * Trigger tests * Add http.statusCode to span before notice_error call * Report number of messages in error trace even if 0 * Revert notice_error and add _nr_message attr * Remove enabled_ml_settings as not needed * Add stats engine _nr_message test * [Mega-Linter] Apply linters fixes * Trigger tests * Revert black formatting in unicode/byte messages --------- Co-authored-by: Uma Annamalai <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: Hannah Stepanek <[email protected]> Co-authored-by: lrafeei <[email protected]> Co-authored-by: hmstepanek <[email protected]> * Bedrock Sync Chat Completion Instrumentation (#953) * Add AWS Bedrock testing infrastructure * Squashed commit of the following: commit 2834663 Author: Timothy Pansino <[email protected]> Date: Mon Oct 9 17:42:05 2023 -0700 OpenAI Mock Backend (#929) * Add mock external openai server * Add mocked OpenAI server fixtures * Set up recorded responses. * Clean mock server to depend on http server * Linting * Pin flask version for flask restx tests. (#931) * Ignore new redis methods. (#932) Co-authored-by: Lalleh Rafeei <[email protected]> * Remove approved paths * Update CI Image (#930) * Update available python versions in CI * Update makefile with overrides * Fix default branch detection for arm builds --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Add mocking for embedding endpoint * [Mega-Linter] Apply linters fixes * Add ratelimit headers * [Mega-Linter] Apply linters fixes * Only get package version once (#928) * Only get package version once * Add disconnect method * Add disconnect method --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Add datalib dependency for embedding testing. * Add OpenAI Test Infrastructure (#926) * Add openai to tox * Add OpenAI test files. * Add test functions. * [Mega-Linter] Apply linters fixes --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: mergify[bot] <mergify[bot]@users.noreply.github.com> * Add mock external openai server * Add mocked OpenAI server fixtures * Set up recorded responses. * Clean mock server to depend on http server * Linting * Remove approved paths * Add mocking for embedding endpoint * [Mega-Linter] Apply linters fixes * Add ratelimit headers * [Mega-Linter] Apply linters fixes * Add datalib dependency for embedding testing. --------- Co-authored-by: Uma Annamalai <[email protected]> Co-authored-by: Lalleh Rafeei <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: TimPansino <[email protected]> Co-authored-by: Hannah Stepanek <[email protected]> Co-authored-by: mergify[bot] <mergify[bot]@users.noreply.github.com> commit db63d45 Author: Uma Annamalai <[email protected]> Date: Mon Oct 2 15:31:38 2023 -0700 Add OpenAI Test Infrastructure (#926) * Add openai to tox * Add OpenAI test files. * Add test functions. * [Mega-Linter] Apply linters fixes --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: mergify[bot] <mergify[bot]@users.noreply.github.com> * Squashed commit of the following: commit 182c7a8 Author: Uma Annamalai <[email protected]> Date: Fri Oct 13 10:12:55 2023 -0700 Add request/ response IDs. commit f6d13f8 Author: Uma Annamalai <[email protected]> Date: Thu Oct 12 13:23:39 2023 -0700 Test cleanup. commit d057663 Author: Uma Annamalai <[email protected]> Date: Tue Oct 10 10:23:00 2023 -0700 Remove commented code. commit dd29433 Author: Uma Annamalai <[email protected]> Date: Tue Oct 10 10:19:01 2023 -0700 Add openai sync instrumentation. commit 2834663 Author: Timothy Pansino <[email protected]> Date: Mon Oct 9 17:42:05 2023 -0700 OpenAI Mock Backend (#929) * Add mock external openai server * Add mocked OpenAI server fixtures * Set up recorded responses. * Clean mock server to depend on http server * Linting * Pin flask version for flask restx tests. (#931) * Ignore new redis methods. (#932) Co-authored-by: Lalleh Rafeei <[email protected]> * Remove approved paths * Update CI Image (#930) * Update available python versions in CI * Update makefile with overrides * Fix default branch detection for arm builds --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Add mocking for embedding endpoint * [Mega-Linter] Apply linters fixes * Add ratelimit headers * [Mega-Linter] Apply linters fixes * Only get package version once (#928) * Only get package version once * Add disconnect method * Add disconnect method --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Add datalib dependency for embedding testing. * Add OpenAI Test Infrastructure (#926) * Add openai to tox * Add OpenAI test files. * Add test functions. * [Mega-Linter] Apply linters fixes --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: mergify[bot] <mergify[bot]@users.noreply.github.com> * Add mock external openai server * Add mocked OpenAI server fixtures * Set up recorded responses. * Clean mock server to depend on http server * Linting * Remove approved paths * Add mocking for embedding endpoint * [Mega-Linter] Apply linters fixes * Add ratelimit headers * [Mega-Linter] Apply linters fixes * Add datalib dependency for embedding testing. --------- Co-authored-by: Uma Annamalai <[email protected]> Co-authored-by: Lalleh Rafeei <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: TimPansino <[email protected]> Co-authored-by: Hannah Stepanek <[email protected]> Co-authored-by: mergify[bot] <mergify[bot]@users.noreply.github.com> commit db63d45 Author: Uma Annamalai <[email protected]> Date: Mon Oct 2 15:31:38 2023 -0700 Add OpenAI Test Infrastructure (#926) * Add openai to tox * Add OpenAI test files. * Add test functions. * [Mega-Linter] Apply linters fixes --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: mergify[bot] <mergify[bot]@users.noreply.github.com> * Cache Package Version Lookups (#946) * Cache _get_package_version * Add Python 2.7 support to get_package_version caching * [Mega-Linter] Apply linters fixes * Bump tests --------- Co-authored-by: SlavaSkvortsov <[email protected]> Co-authored-by: TimPansino <[email protected]> * Fix Redis Generator Methods (#947) * Fix scan_iter for redis * Replace generator methods * Update instance info instrumentation * Remove mistake from uninstrumented methods * Add skip condition to asyncio generator tests * Add skip condition to asyncio generator tests --------- Co-authored-by: Lalleh Rafeei <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * TEMP * Automatic RPM System Updates (#948) * Checkout old action * Adding RPM action * Add dry run * Incorporating action into workflow * Wire secret into custom action * Enable action * Correct action name * Fix syntax * Fix quoting issues * Drop pre-verification. Does not work on python * Fix merge artifact * Bedrock titan extraction nearly complete * Cleaning up titan bedrock implementation * TEMP * Tests for bedrock passing Co-authored-by: Lalleh Rafeei <[email protected]> * Cleaned up titan testing Co-authored-by: Lalleh Rafeei <[email protected]> Co-authored-by: Hannah Stepanek <[email protected]> * Parametrized bedrock testing * Add support for AI21-J2 models * Change to dynamic no conversation id events * Drop all openai refs * [Mega-Linter] Apply linters fixes * Adding response_id and response_model * Drop python 3.7 tests for Hypercorn (#954) * Apply suggestions from code review * Remove unused import --------- Co-authored-by: Uma Annamalai <[email protected]> Co-authored-by: SlavaSkvortsov <[email protected]> Co-authored-by: TimPansino <[email protected]> Co-authored-by: Lalleh Rafeei <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: Lalleh Rafeei <[email protected]> Co-authored-by: Hannah Stepanek <[email protected]> Co-authored-by: Lalleh Rafeei <[email protected]> * Feature bedrock cohere instrumentation (#955) * Add AWS Bedrock testing infrastructure * Squashed commit of the following: commit 2834663 Author: Timothy Pansino <[email protected]> Date: Mon Oct 9 17:42:05 2023 -0700 OpenAI Mock Backend (#929) * Add mock external openai server * Add mocked OpenAI server fixtures * Set up recorded responses. * Clean mock server to depend on http server * Linting * Pin flask version for flask restx tests. (#931) * Ignore new redis methods. (#932) Co-authored-by: Lalleh Rafeei <[email protected]> * Remove approved paths * Update CI Image (#930) * Update available python versions in CI * Update makefile with overrides * Fix default branch detection for arm builds --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Add mocking for embedding endpoint * [Mega-Linter] Apply linters fixes * Add ratelimit headers * [Mega-Linter] Apply linters fixes * Only get package version once (#928) * Only get package version once * Add disconnect method * Add disconnect method --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Add datalib dependency for embedding testing. * Add OpenAI Test Infrastructure (#926) * Add openai to tox * Add OpenAI test files. * Add test functions. * [Mega-Linter] Apply linters fixes --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: mergify[bot] <mergify[bot]@users.noreply.github.com> * Add mock external openai server * Add mocked OpenAI server fixtures * Set up recorded responses. * Clean mock server to depend on http server * Linting * Remove approved paths * Add mocking for embedding endpoint * [Mega-Linter] Apply linters fixes * Add ratelimit headers * [Mega-Linter] Apply linters fixes * Add datalib dependency for embedding testing. --------- Co-authored-by: Uma Annamalai <[email protected]> Co-authored-by: Lalleh Rafeei <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: TimPansino <[email protected]> Co-authored-by: Hannah Stepanek <[email protected]> Co-authored-by: mergify[bot] <mergify[bot]@users.noreply.github.com> commit db63d45 Author: Uma Annamalai <[email protected]> Date: Mon Oct 2 15:31:38 2023 -0700 Add OpenAI Test Infrastructure (#926) * Add openai to tox * Add OpenAI test files. * Add test functions. * [Mega-Linter] Apply linters fixes --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: mergify[bot] <mergify[bot]@users.noreply.github.com> * Squashed commit of the following: commit 182c7a8 Author: Uma Annamalai <[email protected]> Date: Fri Oct 13 10:12:55 2023 -0700 Add request/ response IDs. commit f6d13f8 Author: Uma Annamalai <[email protected]> Date: Thu Oct 12 13:23:39 2023 -0700 Test cleanup. commit d057663 Author: Uma Annamalai <[email protected]> Date: Tue Oct 10 10:23:00 2023 -0700 Remove commented code. commit dd29433 Author: Uma Annamalai <[email protected]> Date: Tue Oct 10 10:19:01 2023 -0700 Add openai sync instrumentation. commit 2834663 Author: Timothy Pansino <[email protected]> Date: Mon Oct 9 17:42:05 2023 -0700 OpenAI Mock Backend (#929) * Add mock external openai server * Add mocked OpenAI server fixtures * Set up recorded responses. * Clean mock server to depend on http server * Linting * Pin flask version for flask restx tests. (#931) * Ignore new redis methods. (#932) Co-authored-by: Lalleh Rafeei <[email protected]> * Remove approved paths * Update CI Image (#930) * Update available python versions in CI * Update makefile with overrides * Fix default branch detection for arm builds --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Add mocking for embedding endpoint * [Mega-Linter] Apply linters fixes * Add ratelimit headers * [Mega-Linter] Apply linters fixes * Only get package version once (#928) * Only get package version once * Add disconnect method * Add disconnect method --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Add datalib dependency for embedding testing. * Add OpenAI Test Infrastructure (#926) * Add openai to tox * Add OpenAI test files. * Add test functions. * [Mega-Linter] Apply linters fixes --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: mergify[bot] <mergify[bot]@users.noreply.github.com> * Add mock external openai server * Add mocked OpenAI server fixtures * Set up recorded responses. * Clean mock server to depend on http server * Linting * Remove approved paths * Add mocking for embedding endpoint * [Mega-Linter] Apply linters fixes * Add ratelimit headers * [Mega-Linter] Apply linters fixes * Add datalib dependency for embedding testing. --------- Co-authored-by: Uma Annamalai <[email protected]> Co-authored-by: Lalleh Rafeei <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: TimPansino <[email protected]> Co-authored-by: Hannah Stepanek <[email protected]> Co-authored-by: mergify[bot] <mergify[bot]@users.noreply.github.com> commit db63d45 Author: Uma Annamalai <[email protected]> Date: Mon Oct 2 15:31:38 2023 -0700 Add OpenAI Test Infrastructure (#926) * Add openai to tox * Add OpenAI test files. * Add test functions. * [Mega-Linter] Apply linters fixes --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: mergify[bot] <mergify[bot]@users.noreply.github.com> * TEMP * Bedrock titan extraction nearly complete * Cleaning up titan bedrock implementation * TEMP * Tests for bedrock passing Co-authored-by: Lalleh Rafeei <[email protected]> * Cleaned up titan testing Co-authored-by: Lalleh Rafeei <[email protected]> Co-authored-by: Hannah Stepanek <[email protected]> * Parametrized bedrock testing * Add support for AI21-J2 models * Change to dynamic no conversation id events * Add cohere model * Remove openai instrumentation from this branch * Remove OpenAI from newrelic/config.py --------- Co-authored-by: Uma Annamalai <[email protected]> Co-authored-by: Tim Pansino <[email protected]> Co-authored-by: Lalleh Rafeei <[email protected]> Co-authored-by: Hannah Stepanek <[email protected]> * AWS Bedrock Embedding Instrumentation (#957) * AWS Bedrock embedding instrumentation * Correct symbol name * Add support for bedrock claude (#960) Co-authored-by: Timothy Pansino <[email protected]> * Combine Botocore Tests (#959) * Initial file migration * Enable DT on all span tests * Add pytest skip for older botocore versions * Fixup: app name merge conflict --------- Co-authored-by: Hannah Stepanek <[email protected]> * Pin openai tests to below 1.0 (#962) * Pin openai below 1.0 * Fixup * Add openai feedback support (#942) * Add get_ai_message_ids & message id capturing * Add tests * Remove generator * Add tests for conversation id unset * Add error code to mocked responses * Remove bedrock tests --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: Uma Annamalai <[email protected]> * Add ingest source to openai events (#961) * Pin openai below 1.0 * Fixup * Add ingest_source to events * Remove duplicate test file * Handle 0.32.0.post1 version in tests (#963) --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Handle 0.32.0.post1 version in tests (#963) * Initial merge commit * Update moto * Test for Bedrock embeddings metrics * Add record_llm_feedback_event API (#964) * Implement record_ai_feedback API. * [Mega-Linter] Apply linters fixes * Change API name to record_ai_feedback_event. * Fix API naming. * Rename to record_llm_feedback_event and get_llm_message_ids. * [Mega-Linter] Apply linters fixes * Address review feedback. * Update test structure. * [Mega-Linter] Apply linters fixes * Bump tests. --------- Co-authored-by: umaannamalai <[email protected]> * Bedrock Error Tracing (#966) * Cache Package Version Lookups (#946) * Cache _get_package_version * Add Python 2.7 support to get_package_version caching * [Mega-Linter] Apply linters fixes * Bump tests --------- Co-authored-by: SlavaSkvortsov <[email protected]> Co-authored-by: TimPansino <[email protected]> * Fix Redis Generator Methods (#947) * Fix scan_iter for redis * Replace generator methods * Update instance info instrumentation * Remove mistake from uninstrumented methods * Add skip condition to asyncio generator tests * Add skip condition to asyncio generator tests --------- Co-authored-by: Lalleh Rafeei <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Automatic RPM System Updates (#948) * Checkout old action * Adding RPM action * Add dry run * Incorporating action into workflow * Wire secret into custom action * Enable action * Correct action name * Fix syntax * Fix quoting issues * Drop pre-verification. Does not work on python * Fix merge artifact * Drop python 3.7 tests for Hypercorn (#954) * Fix pyenv installation for devcontainer (#936) Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Remove duplicate kafka import hook (#956) Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Initial bedrock error tracing commit * Handle 0.32.0.post1 version in tests (#963) * Add status code to mock bedrock server * Updating error response recording logic * Work on bedrock errror tracing * Chat completion error tracing * Adding embedding error tracing * Delete comment * Update moto --------- Co-authored-by: SlavaSkvortsov <[email protected]> Co-authored-by: TimPansino <[email protected]> Co-authored-by: Lalleh Rafeei <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: Lalleh Rafeei <[email protected]> Co-authored-by: Hannah Stepanek <[email protected]> * Fix expected chat completion tests * Remove commented out code * Switch openai to use custom events. * Cleanup. * Switch Bedrock instrumentation to custom events. * Fix record_feedback test. * Fix disabled settings. * Add attribute length setting to bedrock conftest. --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: mergify[bot] <mergify[bot]@users.noreply.github.com> Co-authored-by: Timothy Pansino <[email protected]> Co-authored-by: Lalleh Rafeei <[email protected]> Co-authored-by: TimPansino <[email protected]> Co-authored-by: Hannah Stepanek <[email protected]> Co-authored-by: umaannamalai <[email protected]> Co-authored-by: SlavaSkvortsov <[email protected]> Co-authored-by: Lalleh Rafeei <[email protected]> Co-authored-by: lrafeei <[email protected]> Co-authored-by: hmstepanek <[email protected]> Co-authored-by: Tim Pansino <[email protected]>
newrelic · Nov 15, 2023 · 21730db · 21730db
1 parent ca1d093
commit 21730db
Show file tree

Hide file tree

Showing 12 changed files with 191 additions and 81 deletions.
diff --git a/newrelic/api/ml_model.py b/newrelic/api/ml_model.py
@@ -81,4 +81,4 @@ def record_llm_feedback_event(
     }
     feedback_message_event.update(metadata)
 
-    transaction.record_ml_event("LlmFeedbackMessage", feedback_message_event)
+    transaction.record_custom_event("LlmFeedbackMessage", feedback_message_event)
diff --git a/newrelic/hooks/external_botocore.py b/newrelic/hooks/external_botocore.py
@@ -119,7 +119,7 @@ def create_chat_completion_message_event(
             "vendor": "bedrock",
             "ingest_source": "Python",
         }
-        transaction.record_ml_event("LlmChatCompletionMessage", chat_completion_message_dict)
+        transaction.record_custom_event("LlmChatCompletionMessage", chat_completion_message_dict)
 
 
 def extract_bedrock_titan_text_model(request_body, response_body=None):
@@ -376,7 +376,7 @@ def handle_embedding_event(
         }
     )
 
-    transaction.record_ml_event("LlmEmbedding", embedding_dict)
+    transaction.record_custom_event("LlmEmbedding", embedding_dict)
 
 
 def handle_chat_completion_event(
@@ -413,7 +413,7 @@ def handle_chat_completion_event(
         }
     )
 
-    transaction.record_ml_event("LlmChatCompletionSummary", chat_completion_summary_dict)
+    transaction.record_custom_event("LlmChatCompletionSummary", chat_completion_summary_dict)
 
     create_chat_completion_message_event(
         transaction=transaction,

diff --git a/newrelic/hooks/mlmodel_openai.py b/newrelic/hooks/mlmodel_openai.py
@@ -122,7 +122,8 @@ def wrap_embedding_create(wrapped, instance, args, kwargs):
         "ingest_source": "Python",
     }
 
-    transaction.record_ml_event("LlmEmbedding", embedding_dict)
+    transaction.record_custom_event("LlmEmbedding", embedding_dict)
+
     return response
 
 
@@ -213,7 +214,8 @@ def wrap_chat_completion_create(wrapped, instance, args, kwargs):
         "response.number_of_messages": len(messages) + len(choices),
     }
 
-    transaction.record_ml_event("LlmChatCompletionSummary", chat_completion_summary_dict)
+    transaction.record_custom_event("LlmChatCompletionSummary", chat_completion_summary_dict)
+
     message_list = list(messages)
     if choices:
         message_list.extend([choices[0].message])
@@ -287,7 +289,9 @@ def create_chat_completion_message_event(
             "vendor": "openAI",
             "ingest_source": "Python",
         }
-        transaction.record_ml_event("LlmChatCompletionMessage", chat_completion_message_dict)
+
+        transaction.record_custom_event("LlmChatCompletionMessage", chat_completion_message_dict)
+
     return (conversation_id, request_id, message_ids)
 
 
@@ -368,7 +372,8 @@ async def wrap_embedding_acreate(wrapped, instance, args, kwargs):
         "ingest_source": "Python",
     }
 
-    transaction.record_ml_event("LlmEmbedding", embedding_dict)
+    transaction.record_custom_event("LlmEmbedding", embedding_dict)
+
     return response
 
 
@@ -465,7 +470,8 @@ async def wrap_chat_completion_acreate(wrapped, instance, args, kwargs):
         "ingest_source": "Python",
     }
 
-    transaction.record_ml_event("LlmChatCompletionSummary", chat_completion_summary_dict)
+    transaction.record_custom_event("LlmChatCompletionSummary", chat_completion_summary_dict)
+
     message_list = list(messages)
     if choices:
         message_list.extend([choices[0].message])

diff --git a/tests/agent_features/test_record_llm_feedback_event.py b/tests/agent_features/test_record_llm_feedback_event.py
@@ -12,10 +12,8 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-from testing_support.fixtures import reset_core_stats_engine
-from testing_support.validators.validate_ml_event_count import validate_ml_event_count
-from testing_support.validators.validate_ml_events import validate_ml_events
-
+from testing_support.fixtures import reset_core_stats_engine, validate_custom_event_count
+from testing_support.validators.validate_custom_events import validate_custom_events
 from newrelic.api.background_task import background_task
 from newrelic.api.ml_model import record_llm_feedback_event
 
@@ -38,8 +36,8 @@ def test_record_llm_feedback_event_all_args_supplied():
             },
         ),
     ]
-
-    @validate_ml_events(llm_feedback_all_args_recorded_events)
+    
+    @validate_custom_events(llm_feedback_all_args_recorded_events)
     @background_task()
     def _test():
         record_llm_feedback_event(
@@ -73,7 +71,7 @@ def test_record_llm_feedback_event_required_args_supplied():
         ),
     ]
 
-    @validate_ml_events(llm_feedback_required_args_recorded_events)
+    @validate_custom_events(llm_feedback_required_args_recorded_events)
     @background_task()
     def _test():
         record_llm_feedback_event(message_id="message_id", rating="Good")
@@ -82,7 +80,7 @@ def _test():
 
 
 @reset_core_stats_engine()
-@validate_ml_event_count(count=0)
+@validate_custom_event_count(count=0)
 def test_record_llm_feedback_event_outside_txn():
     record_llm_feedback_event(
         rating="Good",

diff --git a/tests/external_botocore/conftest.py b/tests/external_botocore/conftest.py
@@ -42,7 +42,7 @@
     "transaction_tracer.stack_trace_threshold": 0.0,
     "debug.log_data_collector_payloads": True,
     "debug.record_transaction_failure": True,
-    "ml_insights_events.enabled": True,
+    "custom_insights_events.max_attribute_value": 4096
 }
 collector_agent_registration = collector_agent_registration_fixture(
     app_name="Python Agent Test (external_botocore)",

diff --git a/tests/external_botocore/test_bedrock_chat_completion.py b/tests/external_botocore/test_bedrock_chat_completion.py
@@ -28,12 +28,12 @@
     dt_enabled,
     override_application_settings,
     reset_core_stats_engine,
+    validate_custom_event_count,
 )
 from testing_support.validators.validate_error_trace_attributes import (
     validate_error_trace_attributes,
 )
-from testing_support.validators.validate_ml_event_count import validate_ml_event_count
-from testing_support.validators.validate_ml_events import validate_ml_events
+from testing_support.validators.validate_custom_events import validate_custom_events
 from testing_support.validators.validate_transaction_metrics import (
     validate_transaction_metrics,
 )
@@ -106,9 +106,9 @@ def expected_client_error(model_id):
 # not working with claude
 @reset_core_stats_engine()
 def test_bedrock_chat_completion_in_txn_with_convo_id(set_trace_info, exercise_model, expected_events):
-    @validate_ml_events(expected_events)
+    @validate_custom_events(expected_events)
     # One summary event, one user message, and one response message from the assistant
-    @validate_ml_event_count(count=3)
+    @validate_custom_event_count(count=3)
     @validate_transaction_metrics(
         name="test_bedrock_chat_completion_in_txn_with_convo_id",
         custom_metrics=[
@@ -128,9 +128,9 @@ def _test():
 # not working with claude
 @reset_core_stats_engine()
 def test_bedrock_chat_completion_in_txn_no_convo_id(set_trace_info, exercise_model, expected_events_no_convo_id):
-    @validate_ml_events(expected_events_no_convo_id)
+    @validate_custom_events(expected_events_no_convo_id)
     # One summary event, one user message, and one response message from the assistant
-    @validate_ml_event_count(count=3)
+    @validate_custom_event_count(count=3)
     @validate_transaction_metrics(
         name="test_bedrock_chat_completion_in_txn_no_convo_id",
         custom_metrics=[
@@ -147,19 +147,19 @@ def _test():
 
 
 @reset_core_stats_engine()
-@validate_ml_event_count(count=0)
+@validate_custom_event_count(count=0)
 def test_bedrock_chat_completion_outside_txn(set_trace_info, exercise_model):
     set_trace_info()
     add_custom_attribute("conversation_id", "my-awesome-id")
     exercise_model(prompt=_test_bedrock_chat_completion_prompt, temperature=0.7, max_tokens=100)
 
 
-disabled_ml_settings = {"machine_learning.enabled": False, "ml_insights_events.enabled": False}
+disabled_custom_insights_settings = {"custom_insights_events.enabled": False}
 
 
-@override_application_settings(disabled_ml_settings)
+@override_application_settings(disabled_custom_insights_settings)
 @reset_core_stats_engine()
-@validate_ml_event_count(count=0)
+@validate_custom_event_count(count=0)
 @validate_transaction_metrics(
     name="test_bedrock_chat_completion_disabled_settings",
     custom_metrics=[

diff --git a/tests/external_botocore/test_bedrock_embeddings.py b/tests/external_botocore/test_bedrock_embeddings.py
@@ -27,20 +27,20 @@
     dt_enabled,
     override_application_settings,
     reset_core_stats_engine,
+    validate_custom_event_count
 )
 from testing_support.validators.validate_error_trace_attributes import (
     validate_error_trace_attributes,
 )
-from testing_support.validators.validate_ml_event_count import validate_ml_event_count
-from testing_support.validators.validate_ml_events import validate_ml_events
+from testing_support.validators.validate_custom_events import validate_custom_events
 from testing_support.validators.validate_transaction_metrics import (
     validate_transaction_metrics,
 )
 
 from newrelic.api.background_task import background_task
 from newrelic.common.object_names import callable_name
 
-disabled_ml_insights_settings = {"ml_insights_events.enabled": False}
+disabled_custom_insights_settings = {"custom_insights_events.enabled": False}
 
 
 @pytest.fixture(scope="session", params=[False, True], ids=["Bytes", "Stream"])
@@ -92,8 +92,8 @@ def expected_client_error(model_id):
 
 @reset_core_stats_engine()
 def test_bedrock_embedding(set_trace_info, exercise_model, expected_events):
-    @validate_ml_events(expected_events)
-    @validate_ml_event_count(count=1)
+    @validate_custom_events(expected_events)
+    @validate_custom_event_count(count=1)
     @validate_transaction_metrics(
         name="test_bedrock_embedding",
         custom_metrics=[
@@ -110,7 +110,7 @@ def _test():
 
 
 @reset_core_stats_engine()
-@validate_ml_event_count(count=0)
+@validate_custom_event_count(count=0)
 def test_bedrock_embedding_outside_txn(exercise_model):
     exercise_model(prompt="This is an embedding test.")
 
@@ -119,9 +119,9 @@ def test_bedrock_embedding_outside_txn(exercise_model):
 _client_error_name = callable_name(_client_error)
 
 
-@override_application_settings(disabled_ml_insights_settings)
+@override_application_settings(disabled_custom_insights_settings)
 @reset_core_stats_engine()
-@validate_ml_event_count(count=0)
+@validate_custom_event_count(count=0)
 @validate_transaction_metrics(
     name="test_bedrock_embeddings:test_bedrock_embedding_disabled_settings",
     custom_metrics=[

diff --git a/tests/mlmodel_openai/test_chat_completion.py b/tests/mlmodel_openai/test_chat_completion.py
@@ -16,17 +16,17 @@
 from testing_support.fixtures import (
     override_application_settings,
     reset_core_stats_engine,
+    validate_custom_event_count,
 )
-from testing_support.validators.validate_ml_event_count import validate_ml_event_count
-from testing_support.validators.validate_ml_events import validate_ml_events
+from testing_support.validators.validate_custom_events import validate_custom_events
 from testing_support.validators.validate_transaction_metrics import (
     validate_transaction_metrics,
 )
 
 from newrelic.api.background_task import background_task
 from newrelic.api.transaction import add_custom_attribute
 
-disabled_ml_insights_settings = {"ml_insights_events.enabled": False}
+disabled_custom_insights_settings = {"custom_insights_events.enabled": False}
 
 _test_openai_chat_completion_messages = (
     {"role": "system", "content": "You are a scientist."},
@@ -129,9 +129,9 @@
 
 
 @reset_core_stats_engine()
-@validate_ml_events(chat_completion_recorded_events)
+@validate_custom_events(chat_completion_recorded_events)
 # One summary event, one system message, one user message, and one response message from the assistant
-@validate_ml_event_count(count=4)
+@validate_custom_event_count(count=4)
 @validate_transaction_metrics(
     name="test_chat_completion:test_openai_chat_completion_sync_in_txn_with_convo_id",
     custom_metrics=[
@@ -244,9 +244,9 @@ def test_openai_chat_completion_sync_in_txn_with_convo_id(set_trace_info):
 
 
 @reset_core_stats_engine()
-@validate_ml_events(chat_completion_recorded_events_no_convo_id)
+@validate_custom_events(chat_completion_recorded_events_no_convo_id)
 # One summary event, one system message, one user message, and one response message from the assistant
-@validate_ml_event_count(count=4)
+@validate_custom_event_count(count=4)
 @background_task()
 def test_openai_chat_completion_sync_in_txn_no_convo_id(set_trace_info):
     set_trace_info()
@@ -256,35 +256,35 @@ def test_openai_chat_completion_sync_in_txn_no_convo_id(set_trace_info):
 
 
 @reset_core_stats_engine()
-@validate_ml_event_count(count=0)
+@validate_custom_event_count(count=0)
 def test_openai_chat_completion_sync_outside_txn():
     add_custom_attribute("conversation_id", "my-awesome-id")
     openai.ChatCompletion.create(
         model="gpt-3.5-turbo", messages=_test_openai_chat_completion_messages, temperature=0.7, max_tokens=100
     )
 
 
-@override_application_settings(disabled_ml_insights_settings)
+@override_application_settings(disabled_custom_insights_settings)
 @reset_core_stats_engine()
-@validate_ml_event_count(count=0)
+@validate_custom_event_count(count=0)
 @validate_transaction_metrics(
-    name="test_chat_completion:test_openai_chat_completion_sync_ml_insights_disabled",
+    name="test_chat_completion:test_openai_chat_completion_sync_custom_events_insights_disabled",
     custom_metrics=[
         ("Python/ML/OpenAI/%s" % openai.__version__, 1),
     ],
     background_task=True,
 )
 @background_task()
-def test_openai_chat_completion_sync_ml_insights_disabled(set_trace_info):
+def test_openai_chat_completion_sync_custom_events_insights_disabled(set_trace_info):
     set_trace_info()
     openai.ChatCompletion.create(
         model="gpt-3.5-turbo", messages=_test_openai_chat_completion_messages, temperature=0.7, max_tokens=100
     )
 
 
 @reset_core_stats_engine()
-@validate_ml_events(chat_completion_recorded_events_no_convo_id)
-@validate_ml_event_count(count=4)
+@validate_custom_events(chat_completion_recorded_events_no_convo_id)
+@validate_custom_event_count(count=4)
 @background_task()
 def test_openai_chat_completion_async_conversation_id_unset(loop, set_trace_info):
     set_trace_info()
@@ -297,8 +297,8 @@ def test_openai_chat_completion_async_conversation_id_unset(loop, set_trace_info
 
 
 @reset_core_stats_engine()
-@validate_ml_events(chat_completion_recorded_events)
-@validate_ml_event_count(count=4)
+@validate_custom_events(chat_completion_recorded_events)
+@validate_custom_event_count(count=4)
 @validate_transaction_metrics(
     name="test_chat_completion:test_openai_chat_completion_async_conversation_id_set",
     custom_metrics=[
@@ -319,7 +319,7 @@ def test_openai_chat_completion_async_conversation_id_set(loop, set_trace_info):
 
 
 @reset_core_stats_engine()
-@validate_ml_event_count(count=0)
+@validate_custom_event_count(count=0)
 def test_openai_chat_completion_async_outside_transaction(loop):
     loop.run_until_complete(
         openai.ChatCompletion.acreate(
@@ -328,18 +328,18 @@ def test_openai_chat_completion_async_outside_transaction(loop):
     )
 
 
-@override_application_settings(disabled_ml_insights_settings)
+@override_application_settings(disabled_custom_insights_settings)
 @reset_core_stats_engine()
-@validate_ml_event_count(count=0)
+@validate_custom_event_count(count=0)
 @validate_transaction_metrics(
-    name="test_chat_completion:test_openai_chat_completion_async_disabled_ml_settings",
+    name="test_chat_completion:test_openai_chat_completion_async_disabled_custom_event_settings",
     custom_metrics=[
         ("Python/ML/OpenAI/%s" % openai.__version__, 1),
     ],
     background_task=True,
 )
 @background_task()
-def test_openai_chat_completion_async_disabled_ml_settings(loop):
+def test_openai_chat_completion_async_disabled_custom_event_settings(loop):
     loop.run_until_complete(
         openai.ChatCompletion.acreate(
             model="gpt-3.5-turbo", messages=_test_openai_chat_completion_messages, temperature=0.7, max_tokens=100