rfc: @ls.pytest.mark.parametrize interface

[baskaryan]

almost certainly not handling lazy eval correctly, but what do we think of interface?

• automatically logs a pass/fail feedback based on test pass/fail
• can return whatever other feedback you want as well

@ls.pytest.mark.parametrize("Sample Dataset 3", (lambda x: x))
def test_parametrize(inputs, outputs, reference_outputs) -> list:
    assert inputs == outputs
    return [{"key": "foo", "value": "bar"}]

some example experiments here https://dev.smith.langchain.com/public/e7782ea0-3de5-4352-8cd4-7b2cdbb03e4c/d

[hinthornw]

Things I like about this:

1. Can connect to dataset
2. outputs are fairly localized/transpernt
3. Trace seems sensical (has outputs by default)
4. Parallelized!
5. Think you can re-use the score helping function if you wanted

Things I don't looove about this relative to @unit

1. Seems harder to check multi-step things
2. The actual system is run "outside" the test function
3. Currently seems to be 1 experiment per unit test? Maybe that is the right equivalence though not sure
4. Pytest doesn't like if you return stuff from the test function

[jakerachleff]

How would you set failure conditions? I assume people don't want to actually fail if any evaluation fails?

[jakerachleff]

which might mean allowing customizability on the interface on this

[baskaryan]

this only fails if the actual test raises an error (we need to add a manual pytest.fail for that bc we catch and log all errors in the wrapper L48). so it is customizable by default