testing-sdk

📚 Documentation   •   🖥️ Application   •   🏠 Home

Setup

Install poetry

curl -sSL https://install.python-poetry.org | python3 -

Install Autoblocks CLI

See Autoblocks CLI documentation

Install dependencies

poetry install

Run Autoblocks tests

Set your Autoblocks API key

Retrieve your local testing API key from the settings page and set it as an environment variable:

export AUTOBLOCKS_API_KEY=...

Set your OpenAI API key

export OPENAI_API_KEY=...

Run the tests

npx autoblocks testing exec -m "my first run" -- poetry run start

You should see something like:

You can click on the links next to each test name to dig into more details. You can also find all of your tests on the testing homepage in the Autoblocks application.

GitHub Actions setup

A starter workflow was added in .github/workflows/autoblocks-testing.yml. This workflow runs the tests on every push to the repository and also on a daily schedule.

Repo structure

my_project/
  run.py <-- imports all tests from test_suites/ and runs them
  evaluators/ <-- all common evaluators are implemented here
    some_shared_evaluator1.py
    some_shared_evaluator2.py
  tasks/ <-- all "tasks" are implemented here
    task1.py
    task2.py
  test_suites/ <-- tests for each task
    task1/
      __init__.py <-- implements the runner for task1
      evaluators.py  <-- evaluators used only for task1
      test_cases.py <-- contains test cases for task1
    task2/
      __init__.py <-- implements the runner for task2
      evaluators.py  <-- evaluators used only for task2
      test_cases.py <-- contains test cases for task2

Futher Reading

• Autoblocks Testing documentation