LLM Monitoring Multiple Evals #4718
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Removed bug where automatic permission attachment errors
|
Check out this pull request on See visual diffs & provide feedback on Jupyter Notebooks. Powered by ReviewNB |
AWS CodeBuild CI Report
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository |
AWS CodeBuild CI Report
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository |
AWS CodeBuild CI Report
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository |
AWS CodeBuild CI Report
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository |
AWS CodeBuild CI Report
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository |
sagemaker_model_monitor/llm_multiple_evals_monitor_byoc/src/components/evaluator.py
Show resolved
Hide resolved
AWS CodeBuild CI Report
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository |
nileshvd
approved these changes
Aug 1, 2024
AWS CodeBuild CI Report
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository |
AWS CodeBuild CI Report
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository |
AWS CodeBuild CI Report
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository |
liujiaorr
approved these changes
Aug 2, 2024
zhaoqizqwang
added a commit
that referenced
this pull request
Aug 6, 2024
* Update SMP v2 notebooks to use latest PyTorch 2.3.1, TSM 2.4.0 release (#4678) * Update SMP v2 notebooks to use latest PT2.3.1-TSM2.4.0 release. * Update SMP v2 shared_scripts * Update minimum sagemaker pysdk version to 2.224 * Parsash2 patch 1 (#4690) * tutorials-after-initial-feedback Added descriptive text to make the notebooks stand on their own. * move athena notebook into dedicated folder * renamed athena end2end notebooks * moved pyspark notebook into dedicated directory * minor change: consistent directory naming convention * Added overview, headers, and explantory text Tested the notebook end to end. Added more context for processing jobs and cleaning up. The output is visible in the cells. * Added overview, headers, explanatory text Also added troubleshooting note from further testing. * fix directory locations for new notebooks * clear notebook outputs * added integration for ci test results * updated formatting with black-nb * update athena notebook: fix parse predictions * fixed ci integration for pyspark-etl-training notebook --------- Co-authored-by: Janosch Woschitz <[email protected]> * MLflow setup (#4689) * Add SageMaker MLflow examples * Add badges * Add MLflow setup notebook; upgrade SageMaker Python SDK for deployment notebook * Linting * More linting changes --------- Co-authored-by: Bobby Lindsey <[email protected]> * feat: Model monitor json support for Explainability and Bias (#4696) * initial commit of Blog content: "using step decorator for bedrock fine tuning" (https://sim.amazon.com/issues/ML-16440) (#4657) * initial commit of using step decorator for bedrock fine tuning * ran black command on the notebook * Added CI badges * Added CI badges * fixed typo in notebook title --------- Co-authored-by: Ashish Rawat <[email protected]> Co-authored-by: Zhaoqi <[email protected]> * New folder structure (#4694) * Deleted 17 duplicate notebooks (#4685) * Updated README, removed broken links and fixed markdown (#4687) * New Folder Structure Implementation - Archived remaining geospatial example notebooks (#4691) * Archived remaining geospatial example notebooks * Removed geospatial from README.md * Archived remaining workshop notebooks (#4692) * Archived outdated example notebooks between 1-90 views (#4693) --------- Co-authored-by: jsmul <[email protected]> * Revert "New folder structure (#4694)" (#4701) This reverts commit 970d88e due to broken blog links * adding notebook for forecast to canvas workshop (#4704) * adding notebook for forecast to canvas workshop * formatting the notebook using black * Adds notebook for deploying and monitoring llm on sagemaker usin fmeval for evaluation (#4705) Co-authored-by: Brent Friedman <[email protected]> * Update athena_ml_workflow_end_to_end.ipynb (#4702) Fixed bucket names and external links. No change to underlying code or formatting. Co-authored-by: sage-maker <[email protected]> * SageMaker FasterAutoscaling Llama3-8B TGI, real-time endpoints (#4712) * SageMaker FasterAutoscaling Llama3-8B TGI, real-time endpoints * Moved trigger autoscaling to shell script. Removed shell=True in subprocess.Popen --------- Co-authored-by: Aditi Sharma <[email protected]> * Removed bug where automatic permission attachment errors (#4714) * new file structure: re-added pull request template and files needed for rtd build * SageMaker pipeline step decorator blog (#4715) * initial commit of using step decorator for bedrock fine tuning * ran black command on the notebook * Added CI badges * Added CI badges * fixed typo in notebook title * added comments and reviewer feedback --------- Co-authored-by: Ashish Rawat <[email protected]> Co-authored-by: Zhaoqi <[email protected]> * new file structure: re-added RTD files, finished archiving/organizing, added READMEs to folders * new file structure: updated index.rst with new file structure / archived one notebook, renamed one notebook * new file structure: updated contributing.md * LLM Monitoring Multiple Evals (#4718) * Removed bug where automatic permission attachment errors * Adds notebook for monitoring llm with multiple eval libraries --------- Co-authored-by: Brent Friedman <[email protected]> Co-authored-by: nileshvd <[email protected]> * new file structure: updated badging path for end-to-end examples * new file structure: updated badging path for prepare data notebooks * new file structure: updated badging path for build and train models folder * new file structure: updated badging path for deploy and monitor folder * new file structure: updated badging path for generative ai folder * new file structure: updated badging path for ml ops folder * new file structure: updated badging path for responsible ai folder * new file structure: updated folder READMEs, and added several notebook titles * new folder structure: updated pull request template * new file structure: updated readme with announcement, added new notebook naming excel sheet --------- Co-authored-by: Victor Zhu <[email protected]> Co-authored-by: parsash2 <[email protected]> Co-authored-by: Janosch Woschitz <[email protected]> Co-authored-by: Bobby Lindsey <[email protected]> Co-authored-by: Bobby Lindsey <[email protected]> Co-authored-by: zicanl-amazon <[email protected]> Co-authored-by: ashrawat <[email protected]> Co-authored-by: Ashish Rawat <[email protected]> Co-authored-by: Zhaoqi <[email protected]> Co-authored-by: pro-biswa <[email protected]> Co-authored-by: brentfriedman725 <[email protected]> Co-authored-by: Brent Friedman <[email protected]> Co-authored-by: sage-maker <[email protected]> Co-authored-by: Praveen Chamarthi <[email protected]> Co-authored-by: Aditi Sharma <[email protected]> Co-authored-by: nileshvd <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Issue #, if available:
Description of changes:
Testing done:
Merge Checklist
Put an
xin the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your pull request.black-nb -l 100 {path}/{notebook-name}.ipynbBy submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.