Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: create an azure-ml pipeline for eval_prompts() #36

Merged
merged 6 commits into from
Feb 7, 2024
Merged
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 32 additions & 0 deletions azureml/eval_prompts.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
$schema: https://azuremlschemas.azureedge.net/latest/commandJob.schema.json
command: >
python -m autora.doc.pipelines.main eval-prompts
${{inputs.data_dir}}/data.jsonl
${{inputs.data_dir}}/all_prompt.json
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's make the prompts file its own input parameter. This will make it easier to run experiments.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should I parameterize the data file as well? The default value can be 'data.jsonl' but a user can modify input if they want. WDYT?

--model-path ${{inputs.model_path}}
--param do_sample=${{inputs.do_sample}}
--param temperature=${{inputs.temperature}}
--param top_k=${{inputs.top_k}}
--param top_p=${{inputs.top_p}}
code: ../src
inputs:
data_dir:
type: uri_folder
path: azureml://datastores/workspaceblobstore/paths/data/sweetpea/
# Currently models are loading faster directly from HuggingFace vs Azure Blob Storage
# model_dir:
# type: uri_folder
# path: azureml://datastores/workspaceblobstore/paths/base_models
model_path: meta-llama/Llama-2-7b-chat-hf
temperature: 0.01
do_sample: 0
top_p: 0.95
top_k: 1
# using a curated environment doesn't work because we need additional packages
environment: # azureml://registries/azureml/environments/acpt-pytorch-2.0-cuda11.7/versions/21
image: mcr.microsoft.com/azureml/curated/acpt-pytorch-2.0-cuda11.7:21
conda_file: conda.yml
display_name: autodoc_multi_prompts_prediction
compute: azureml:v100cluster
experiment_name: evaluation_multi_prompts
description: Run code-to-documentation generation on data_file for each prompt in prompts_file
Loading