Updated ParameterizedModelParser run method for handing run_with_dependencies parameter #923

sp6370 · 2024-01-14T21:13:48Z

Context: Currently ParameterizedModelParser run method utilized kwargs to decide if to execute prompt with dependencies or not. This pull request eliminates the usage of kwargs and introduces a new optional prameter for run method to pass information about running the prompt with dependencies.

Closes: #664

Test Plan:

Add a new test that invokes run with run_with_dependencies=True for a prompt which requires output from another prompt.
Ensure that all the prompts in the hugging face cookbook work.

…th_dependencies parameter

sp6370 · 2024-01-14T21:17:34Z

Hi @rossdanlm!

Could I get some feedback on the test for the PR?

python/tests/test_run_config.py

python/tests/aiconfigs/tarvel_gpt_prompts_with_dependency.json

jonathanlastmileai

Thanks for the PR! Back to your queue for now

python/src/aiconfig/default_parsers/parameterized_model_parser.py

python/tests/aiconfigs/tarvel_gpt_prompts_with_dependency.json

jonathanlastmileai · 2024-01-31T20:56:13Z

python/tests/test_run_config.py

@@ -48,3 +50,23 @@ async def test_load_parametrized_data_config(set_temporary_env_vars):
                },
            ],
        }
+
+@pytest.mark.xfail


Why are we adding a test we expect to fail?

As mentioned by @Ankush-lastmile here. There have been changes to OpenAI model paeser as such new way to for Mocking OpenAI API is required.

The new test introduced works fines based on the earlier mocking approach. Unless a mocking strategy is implemented we would have to use the work around as used here as well https://github.com/sp6370/aiconfig/blob/9617bd9d54e3bff3bb7eef2889af9f30adf39602/python/tests/test_run_config.py#L11.

I think it's good to add, we can edit it later

I have e-enabled the test.

rossdanlm

I think this is good, thanks for the fix, pls ping me when this lands!

python/src/aiconfig/default_parsers/parameterized_model_parser.py

rossdanlm · 2024-02-05T17:56:15Z

python/tests/test_run_config.py

@@ -48,3 +50,23 @@ async def test_load_parametrized_data_config(set_temporary_env_vars):
                },
            ],
        }
+
+@pytest.mark.xfail


I think it's good to add, we can edit it later

rossdanlm

Actually wait sorry, can you pls also remove kwargs from

aiconfig/python/src/aiconfig/model_parser.py

Line 69 in 9617bd9

    
           **kwargs,  # TODO: Remove this, just a hack for now to ensure that it doesn't break

?

See more comments and details in #882

sp6370 · 2024-02-11T20:17:21Z

Actually wait sorry, can you pls also remove kwargs from

aiconfig/python/src/aiconfig/model_parser.py

Line 69 in 9617bd9

**kwargs, # TODO: Remove this, just a hack for now to ensure that it doesn't break

?
See more comments and details in #882

Hi @rossdanlm!
I have addressed this. Also, here's a short video for testing Hugging Face Functionality.
https://www.loom.com/share/a7e09d403dcd4d30a7fd93f8a9e8b7c5

This is GitHub issue for ffmpeg missing error.
#1154

python/src/aiconfig/model_parser.py

rossdanlm

Pls ensure that when testing it's linked to the local pacakge

…encies instead of **kwargs for run

…cies for run method

python/src/aiconfig/Config.py

rossdanlm · 2024-03-20T20:57:16Z

python/tests/conftest.py

+    for input_params, response in response_map_list:
+        if kwargs == input_params:
+            return response
+    raise Exception("Unexpected arguments:\n {}".format(kwargs))


Probably shoudl remove now to show that it works now that kwargs is removed

python/tests/conftest.py

rossdanlm

Generally lgtm, just make sure that we've chcked all the default model parsers too:

aiconfig/python/src/aiconfig/default_parsers/.*

sp6370 · 2024-03-20T21:29:49Z

Cross checked the default model parsers aiconfig/python/src/aiconfig/default_parsers/.*. They don't have any implementations of run which requires update.

rossdanlm · 2024-03-20T21:54:58Z

python/tests/test_run_config.py

+        new=mock_openai,
+    ):
+        config_relative_path = (
+            "aiconfigs/tarvel_gpt_prompts_with_dependency.json"


nit: travel typo

rossdanlm

Nice! Thanks for the fixes! I'll follow up later with extensions + publishing requirements!

This is from a while ago, dismissing to land this

sp6370 · 2024-03-21T14:57:22Z

@rossdancraig I share the info on the test plan later today.

sp6370 · 2024-03-22T16:38:16Z

Tests:

Hugging Face Local Inference: https://www.loom.com/share/256142de11e147d19640ce5fd075383b?sid=546a14da-5907-4b19-a9ae-5b97de16e543
Hugging Face Remote Inference: https://www.loom.com/share/cb99fab92e7a47f78aae09906ef3f44a?sid=7c71e5d7-28da-4e0f-aa01-065db9085a04

rossdanlm · 2024-03-27T19:00:10Z

Tests:

Hugging Face Local Inference: https://www.loom.com/share/256142de11e147d19640ce5fd075383b?sid=546a14da-5907-4b19-a9ae-5b97de16e543

Hugging Face Remote Inference: https://www.loom.com/share/cb99fab92e7a47f78aae09906ef3f44a?sid=7c71e5d7-28da-4e0f-aa01-065db9085a04

Thank you for the super detailed test plans!!!

sp6370 added 2 commits January 14, 2024 16:02

chore: Updated ParameterizedModelParser run method for handing run_wi…

739f9b4

…th_dependencies parameter

test: Add test for running a prompt with dependencies

f5ef8ce

sp6370 added 7 commits January 25, 2024 01:53

modified run with dep tests

57a02a7

Added new aiconfig for tests

5f4c489

Added new test for run with dependencies

2eade61

Update openai.py

a4d2024

Update openai.py

d365e07

Update tarvel_gpt_prompts_with_dependency.json

75de4f6

Update test_run_config.py

3af8ac0

sp6370 marked this pull request as ready for review January 27, 2024 21:39

sp6370 requested review from saqadri, rholinshead, suyoglastmileai, Ankush-lastmile, jonathanlastmileai and rossdanlm as code owners January 27, 2024 21:39

sp6370 force-pushed the main branch from da3a2b0 to 3af8ac0 Compare January 27, 2024 21:49

sp6370 changed the title ~~[WIP] Updated ParameterizedModelParser run method for handing run_with_dependencies parameter~~ Updated ParameterizedModelParser run method for handing run_with_dependencies parameter Jan 27, 2024

Ankush-lastmile reviewed Jan 29, 2024

View reviewed changes

python/tests/test_run_config.py Show resolved Hide resolved

Ankush-lastmile reviewed Jan 31, 2024

View reviewed changes

python/tests/aiconfigs/tarvel_gpt_prompts_with_dependency.json Outdated Show resolved Hide resolved

sp6370 added 2 commits January 31, 2024 14:37

Merge branch 'main' into main

39e5986

Update test_run_config.py

9617bd9

jonathanlastmileai previously requested changes Jan 31, 2024

View reviewed changes

rossdanlm approved these changes Feb 5, 2024

View reviewed changes

rossdanlm suggested changes Feb 6, 2024

View reviewed changes

sp6370 added 4 commits February 6, 2024 17:56

Merge branch 'lastmile-ai:main' into main

9df553d

Merge branch 'lastmile-ai:main' into main

9546fbd

Merge branch 'lastmile-ai:main' into main

150257e

Remove unnecessary kwargs hack in ModelParser constructor

e1267db

sp6370 requested a review from rossdanlm February 12, 2024 17:59

rossdanlm reviewed Feb 12, 2024

View reviewed changes

python/src/aiconfig/model_parser.py Show resolved Hide resolved

rossdanlm suggested changes Feb 12, 2024

View reviewed changes

sp6370 added 10 commits February 25, 2024 12:04

Merge branch 'lastmile-ai:main' into main

a93af45

Test Commit

a8d29b8

Remove unnecessary word in FAQ

fa167d5

Merge branch 'lastmile-ai:main' into main

5ae5c47

chore: fixed formatting issues

68a875e

chore: updates the ModelParser to not use **kwargs

2221959

chore: updates the AiConfig runtime to explicitly use run_with_depend…

cc6ee3e

…encies instead of **kwargs for run

chore: eliminates the usage of **kwargs in favor of run_with_dependen…

88d1f50

…cies for run method

chore: fix linting

1a4a8c9

chore: fix formatting

4b5c936

rossdanlm reviewed Mar 20, 2024

View reviewed changes

python/src/aiconfig/Config.py Show resolved Hide resolved

rossdanlm reviewed Mar 20, 2024

View reviewed changes

python/tests/conftest.py Show resolved Hide resolved

rossdanlm reviewed Mar 20, 2024

View reviewed changes

chore: make all callsites consistent

b495e2d

test: enables the test for prompt with dependencies

1ecc940

sp6370 requested a review from rossdancraig March 20, 2024 21:52

rossdanlm reviewed Mar 20, 2024

View reviewed changes

rossdanlm approved these changes Mar 20, 2024

View reviewed changes

chore: fix the typo

d61fa62

rossdanlm merged commit a11d07a into lastmile-ai:main Mar 27, 2024
2 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Updated ParameterizedModelParser run method for handing run_with_dependencies parameter #923

Updated ParameterizedModelParser run method for handing run_with_dependencies parameter #923

sp6370 commented Jan 14, 2024 •

edited

Loading

sp6370 commented Jan 14, 2024

jonathanlastmileai left a comment

jonathanlastmileai Jan 31, 2024

sp6370 Feb 1, 2024

rossdanlm Feb 5, 2024

sp6370 Mar 20, 2024

rossdanlm left a comment

rossdanlm Feb 5, 2024

rossdanlm left a comment

sp6370 commented Feb 11, 2024 •

edited

Loading

rossdanlm left a comment

rossdanlm Mar 20, 2024

rossdanlm left a comment

sp6370 commented Mar 20, 2024

rossdanlm Mar 20, 2024

rossdanlm left a comment

sp6370 commented Mar 21, 2024

sp6370 commented Mar 22, 2024

rossdanlm commented Mar 27, 2024

Updated ParameterizedModelParser run method for handing run_with_dependencies parameter #923

Updated ParameterizedModelParser run method for handing run_with_dependencies parameter #923

Conversation

sp6370 commented Jan 14, 2024 • edited Loading

sp6370 commented Jan 14, 2024

jonathanlastmileai left a comment

Choose a reason for hiding this comment

jonathanlastmileai Jan 31, 2024

Choose a reason for hiding this comment

sp6370 Feb 1, 2024

Choose a reason for hiding this comment

rossdanlm Feb 5, 2024

Choose a reason for hiding this comment

sp6370 Mar 20, 2024

Choose a reason for hiding this comment

rossdanlm left a comment

Choose a reason for hiding this comment

rossdanlm Feb 5, 2024

Choose a reason for hiding this comment

rossdanlm left a comment

Choose a reason for hiding this comment

sp6370 commented Feb 11, 2024 • edited Loading

rossdanlm left a comment

Choose a reason for hiding this comment

rossdanlm Mar 20, 2024

Choose a reason for hiding this comment

rossdanlm left a comment

Choose a reason for hiding this comment

sp6370 commented Mar 20, 2024

rossdanlm Mar 20, 2024

Choose a reason for hiding this comment

rossdanlm left a comment

Choose a reason for hiding this comment

sp6370 commented Mar 21, 2024

sp6370 commented Mar 22, 2024

rossdanlm commented Mar 27, 2024

sp6370 commented Jan 14, 2024 •

edited

Loading

sp6370 commented Feb 11, 2024 •

edited

Loading