[fine-tuning] Integrate Ray benchmarking as an alternative fine-tuning job #580

kpouget · 2024-11-05T15:04:11Z

No description provided.

openshift-ci · 2024-11-05T15:04:17Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please ask for approval from kpouget. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

topsail-bot · 2024-11-05T15:27:52Z

Jenkins Job #1596

🔴 Test of 'rhoai test test_ci' failed after 00 hours 03 minutes 09 seconds. 🔴

• Link to the test results.

• Link to the reports index.

Test configuration:

# RHOAI: run fine_tuning test test_ci
PR_POSITIONAL_ARGS: ray
PR_POSITIONAL_ARG_0: fine_tuning-perf-ci
PR_POSITIONAL_ARG_1: ray

• Link to the Rebuild page.

Failure indicator:

/logs/artifacts/000_test_ci/002__test_fine_tuning/FAILURE | CalledProcessError: Command 'set -o errexit;set -o pipefail;set -o nounset;set -o errtrace;ARTIFACT_DIR="/logs/artifacts/000_test_ci/002__test_fine_tuning" ./run_toolbox.py from_config fine_tuning ray_fine_tuning_job --extra="{'name': 'fine-tuning', 'model_name': 'bigscience/bloom-560m@hf', 'dataset_name': 'twitter_complaints_small.json', 'gpu': 1, 'dataset_replication': 1}"' returned non-zero exit status 1.
Traceback (most recent call last):
  File "/opt/topsail/src/projects/fine_tuning/testing/test_finetuning.py", line 132, in _run_test
    run.run_toolbox_from_config(
  File "/opt/topsail/src/projects/core/library/run.py", line 49, in run_toolbox_from_config
    return run(f'{cmd_env} ./run_toolbox.py from_config {group} {command} {_dict_to_run_toolbox_args(kwargs)}', **run_kwargs)
  File "/opt/topsail/src/projects/core/library/run.py", line 105, in run
    proc = subprocess.run(command, **args)
  File "/usr/lib64/python3.9/subprocess.py", line 528, in run
    raise CalledProcessError(retcode, process.args,

[...]

[Test ran on the internal Perflab CI]

topsail-bot · 2024-11-05T15:40:08Z

Jenkins Job #1597

🔴 Test of 'rhoai test test_ci' failed after 00 hours 00 minutes 04 seconds. 🔴

• Link to the test results.

• Link to the reports index.

Test configuration:

# RHOAI: run fine_tuning test test_ci
PR_POSITIONAL_ARGS: ray
PR_POSITIONAL_ARG_0: fine_tuning-perf-ci
PR_POSITIONAL_ARG_1: ray

• Link to the Rebuild page.

Failure indicator:

/logs/artifacts/000_test_ci/FAILURE | Traceback (most recent call last):
  File "/opt/topsail/src/projects/fine_tuning/testing/test_finetuning.py", line 368, in test
    failed = _run_test_and_visualize()
  File "/opt/topsail/src/projects/fine_tuning/testing/test_finetuning.py", line 262, in _run_test_and_visualize
    raise RuntimeError(msg)
RuntimeError: RHOAI not installed, cluster not prepared for fine-tuning

[Test ran on the internal Perflab CI]

topsail-bot · 2024-11-05T15:58:13Z

Jenkins Job #1598

🔴 Test of 'rhoai test test_ci' failed after 00 hours 04 minutes 35 seconds. 🔴

• Link to the test results.

• Link to the reports index.

Test configuration:

# RHOAI: run fine_tuning test test_ci
PR_POSITIONAL_ARGS: ray
PR_POSITIONAL_ARG_0: fine_tuning-perf-ci
PR_POSITIONAL_ARG_1: ray

• Link to the Rebuild page.

Failure indicator:

/logs/artifacts/000_test_ci/002__test_fine_tuning/000__fine_tuning__ray_fine_tuning_job/FAILURE | [000__fine_tuning__ray_fine_tuning_job] ./run_toolbox.py from_config fine_tuning ray_fine_tuning_job --extra={'name': 'fine-tuning', 'model_name': 'bigscience/bloom-560m@hf', 'dataset_name': 'twitter_complaints_small.json', 'gpu': 1, 'dataset_replication': 1} --> 2
/logs/artifacts/000_test_ci/002__test_fine_tuning/FAILURE | CalledProcessError: Command 'set -o errexit;set -o pipefail;set -o nounset;set -o errtrace;ARTIFACT_DIR="/logs/artifacts/000_test_ci/002__test_fine_tuning" ./run_toolbox.py from_config fine_tuning ray_fine_tuning_job --extra="{'name': 'fine-tuning', 'model_name': 'bigscience/bloom-560m@hf', 'dataset_name': 'twitter_complaints_small.json', 'gpu': 1, 'dataset_replication': 1}"' returned non-zero exit status 2.
Traceback (most recent call last):
  File "/opt/topsail/src/projects/fine_tuning/testing/test_finetuning.py", line 132, in _run_test
    run.run_toolbox_from_config(
  File "/opt/topsail/src/projects/core/library/run.py", line 49, in run_toolbox_from_config
    return run(f'{cmd_env} ./run_toolbox.py from_config {group} {command} {_dict_to_run_toolbox_args(kwargs)}', **run_kwargs)
  File "/opt/topsail/src/projects/core/library/run.py", line 105, in run
    proc = subprocess.run(command, **args)
  File "/usr/lib64/python3.9/subprocess.py", line 528, in run

[...]

[Test ran on the internal Perflab CI]

topsail-bot · 2024-11-05T16:12:45Z

Jenkins Job #1599

🔴 Test of 'rhoai test test_ci' failed after 00 hours 00 minutes 04 seconds. 🔴

• Link to the test results.

• Link to the reports index.

Test configuration:

# RHOAI: run fine_tuning test test_ci
PR_POSITIONAL_ARGS: ray
PR_POSITIONAL_ARG_0: fine_tuning-perf-ci
PR_POSITIONAL_ARG_1: ray

• Link to the Rebuild page.

Failure indicator:

/logs/artifacts/000_test_ci/FAILURE | Traceback (most recent call last):
  File "/opt/topsail/src/projects/fine_tuning/testing/test_finetuning.py", line 368, in test
    failed = _run_test_and_visualize()
  File "/opt/topsail/src/projects/fine_tuning/testing/test_finetuning.py", line 262, in _run_test_and_visualize
    raise RuntimeError(msg)
RuntimeError: RHOAI not installed, cluster not prepared for fine-tuning

[Test ran on the internal Perflab CI]

topsail-bot · 2024-11-05T20:24:22Z

Jenkins Job #1600

🔴 Test of 'rhoai test test_ci' failed after 00 hours 04 minutes 22 seconds. 🔴

• Link to the test results.

• Link to the reports index.

Test configuration:

# RHOAI: run fine_tuning test test_ci
PR_POSITIONAL_ARGS: ray
PR_POSITIONAL_ARG_0: fine_tuning-perf-ci
PR_POSITIONAL_ARG_1: ray

• Link to the Rebuild page.

Failure indicator:

/logs/artifacts/000_test_ci/002__test_fine_tuning/000__fine_tuning__ray_fine_tuning_job/FAILURE | [000__fine_tuning__ray_fine_tuning_job] ./run_toolbox.py from_config fine_tuning ray_fine_tuning_job --extra={'name': 'fine-tuning', 'model_name': 'bigscience/bloom-560m@hf', 'dataset_name': 'twitter_complaints_small.json', 'gpu': 1, 'dataset_replication': 1} --> 2
/logs/artifacts/000_test_ci/002__test_fine_tuning/FAILURE | CalledProcessError: Command 'set -o errexit;set -o pipefail;set -o nounset;set -o errtrace;ARTIFACT_DIR="/logs/artifacts/000_test_ci/002__test_fine_tuning" ./run_toolbox.py from_config fine_tuning ray_fine_tuning_job --extra="{'name': 'fine-tuning', 'model_name': 'bigscience/bloom-560m@hf', 'dataset_name': 'twitter_complaints_small.json', 'gpu': 1, 'dataset_replication': 1}"' returned non-zero exit status 2.
Traceback (most recent call last):
  File "/opt/topsail/src/projects/fine_tuning/testing/test_finetuning.py", line 132, in _run_test
    run.run_toolbox_from_config(
  File "/opt/topsail/src/projects/core/library/run.py", line 49, in run_toolbox_from_config
    return run(f'{cmd_env} ./run_toolbox.py from_config {group} {command} {_dict_to_run_toolbox_args(kwargs)}', **run_kwargs)
  File "/opt/topsail/src/projects/core/library/run.py", line 105, in run
    proc = subprocess.run(command, **args)
  File "/usr/lib64/python3.9/subprocess.py", line 528, in run

[...]

[Test ran on the internal Perflab CI]

topsail-bot · 2024-11-05T20:49:59Z

Jenkins Job #1601

🔴 Test of 'rhoai test test_ci' failed after 00 hours 05 minutes 35 seconds. 🔴

• Link to the test results.

• Link to the reports index.

Test configuration:

# RHOAI: run fine_tuning test test_ci
PR_POSITIONAL_ARGS: ray
PR_POSITIONAL_ARG_0: fine_tuning-perf-ci
PR_POSITIONAL_ARG_1: ray

• Link to the Rebuild page.

Failure indicator:

/logs/artifacts/000_test_ci/002__test_fine_tuning/000__fine_tuning__ray_fine_tuning_job/FAILURE | [000__fine_tuning__ray_fine_tuning_job] ./run_toolbox.py from_config fine_tuning ray_fine_tuning_job --extra={'name': 'fine-tuning', 'model_name': 'bigscience/bloom-560m@hf', 'dataset_name': 'twitter_complaints_small.json', 'gpu': 1, 'dataset_replication': 1} --> 2
/logs/artifacts/000_test_ci/002__test_fine_tuning/FAILURE | CalledProcessError: Command 'set -o errexit;set -o pipefail;set -o nounset;set -o errtrace;ARTIFACT_DIR="/logs/artifacts/000_test_ci/002__test_fine_tuning" ./run_toolbox.py from_config fine_tuning ray_fine_tuning_job --extra="{'name': 'fine-tuning', 'model_name': 'bigscience/bloom-560m@hf', 'dataset_name': 'twitter_complaints_small.json', 'gpu': 1, 'dataset_replication': 1}"' returned non-zero exit status 2.
Traceback (most recent call last):
  File "/opt/topsail/src/projects/fine_tuning/testing/test_finetuning.py", line 132, in _run_test
    run.run_toolbox_from_config(
  File "/opt/topsail/src/projects/core/library/run.py", line 49, in run_toolbox_from_config
    return run(f'{cmd_env} ./run_toolbox.py from_config {group} {command} {_dict_to_run_toolbox_args(kwargs)}', **run_kwargs)
  File "/opt/topsail/src/projects/core/library/run.py", line 105, in run
    proc = subprocess.run(command, **args)
  File "/usr/lib64/python3.9/subprocess.py", line 528, in run

[...]

[Test ran on the internal Perflab CI]

topsail-bot · 2024-11-05T21:37:10Z

Jenkins Job #1602

🔴 Test of 'rhoai test test_ci' failed after 00 hours 25 minutes 02 seconds. 🔴

• Link to the test results.

• Link to the reports index.

Test configuration:

# RHOAI: run fine_tuning test test_ci
PR_POSITIONAL_ARGS: ray
PR_POSITIONAL_ARG_0: fine_tuning-perf-ci
PR_POSITIONAL_ARG_1: ray

• Link to the Rebuild page.

Failure indicator:

/logs/artifacts/000_test_ci/002__test_fine_tuning/000__fine_tuning__ray_fine_tuning_job/FAILURE | [000__fine_tuning__ray_fine_tuning_job] ./run_toolbox.py from_config fine_tuning ray_fine_tuning_job --extra={'name': 'fine-tuning', 'model_name': 'bigscience/bloom-560m@hf', 'dataset_name': 'twitter_complaints_small.json', 'gpu': 1, 'dataset_replication': 1} --> 2
/logs/artifacts/000_test_ci/002__test_fine_tuning/FAILURE | CalledProcessError: Command 'set -o errexit;set -o pipefail;set -o nounset;set -o errtrace;ARTIFACT_DIR="/logs/artifacts/000_test_ci/002__test_fine_tuning" ./run_toolbox.py from_config fine_tuning ray_fine_tuning_job --extra="{'name': 'fine-tuning', 'model_name': 'bigscience/bloom-560m@hf', 'dataset_name': 'twitter_complaints_small.json', 'gpu': 1, 'dataset_replication': 1}"' returned non-zero exit status 2.
Traceback (most recent call last):
  File "/opt/topsail/src/projects/fine_tuning/testing/test_finetuning.py", line 132, in _run_test
    run.run_toolbox_from_config(
  File "/opt/topsail/src/projects/core/library/run.py", line 49, in run_toolbox_from_config
    return run(f'{cmd_env} ./run_toolbox.py from_config {group} {command} {_dict_to_run_toolbox_args(kwargs)}', **run_kwargs)
  File "/opt/topsail/src/projects/core/library/run.py", line 105, in run
    proc = subprocess.run(command, **args)
  File "/usr/lib64/python3.9/subprocess.py", line 528, in run

[...]

[Test ran on the internal Perflab CI]

topsail-bot · 2024-11-06T08:31:52Z

Jenkins Job #1605

🔴 Test of 'rhoai test test_ci' failed after 00 hours 00 minutes 05 seconds. 🔴

• Link to the test results.

• Link to the reports index.

Test configuration:

# RHOAI: run fine_tuning test test_ci
PR_POSITIONAL_ARGS: ray
PR_POSITIONAL_ARG_0: fine_tuning-perf-ci
PR_POSITIONAL_ARG_1: ray

• Link to the Rebuild page.

Failure indicator:

/logs/artifacts/000_test_ci/FAILURE | Traceback (most recent call last):
  File "/opt/topsail/src/projects/fine_tuning/testing/test_finetuning.py", line 368, in test
    failed = _run_test_and_visualize()
  File "/opt/topsail/src/projects/fine_tuning/testing/test_finetuning.py", line 262, in _run_test_and_visualize
    raise RuntimeError(msg)
RuntimeError: RHOAI not installed, cluster not prepared for fine-tuning

[Test ran on the internal Perflab CI]

topsail-bot · 2024-11-06T09:16:35Z

Jenkins Job #1606

🔴 Test of 'rhoai test test_ci' failed after 00 hours 10 minutes 01 seconds. 🔴

• Link to the test results.

• Link to the reports index.

Test configuration:

# RHOAI: run fine_tuning test test_ci
PR_POSITIONAL_ARGS: ray
PR_POSITIONAL_ARG_0: fine_tuning-perf-ci
PR_POSITIONAL_ARG_1: ray

• Link to the Rebuild page.

Failure indicator:

/logs/artifacts/001_test_ci/002__test_fine_tuning/000__fine_tuning__ray_fine_tuning_job/FAILURE | [000__fine_tuning__ray_fine_tuning_job] ./run_toolbox.py from_config fine_tuning ray_fine_tuning_job --extra={'name': 'fine-tuning', 'model_name': 'bigscience/bloom-560m@hf', 'dataset_name': 'twitter_complaints_small.json', 'gpu': 1, 'dataset_replication': 1} --> 2
/logs/artifacts/001_test_ci/002__test_fine_tuning/FAILURE | CalledProcessError: Command 'set -o errexit;set -o pipefail;set -o nounset;set -o errtrace;ARTIFACT_DIR="/logs/artifacts/001_test_ci/002__test_fine_tuning" ./run_toolbox.py from_config fine_tuning ray_fine_tuning_job --extra="{'name': 'fine-tuning', 'model_name': 'bigscience/bloom-560m@hf', 'dataset_name': 'twitter_complaints_small.json', 'gpu': 1, 'dataset_replication': 1}"' returned non-zero exit status 2.
Traceback (most recent call last):
  File "/opt/topsail/src/projects/fine_tuning/testing/test_finetuning.py", line 132, in _run_test
    run.run_toolbox_from_config(
  File "/opt/topsail/src/projects/core/library/run.py", line 49, in run_toolbox_from_config
    return run(f'{cmd_env} ./run_toolbox.py from_config {group} {command} {_dict_to_run_toolbox_args(kwargs)}', **run_kwargs)
  File "/opt/topsail/src/projects/core/library/run.py", line 105, in run
    proc = subprocess.run(command, **args)
  File "/usr/lib64/python3.9/subprocess.py", line 528, in run

[...]

[Test ran on the internal Perflab CI]

topsail-bot · 2024-11-06T09:46:30Z

Jenkins Job #1607

🔴 Test of 'rhoai test test_ci' failed after 00 hours 09 minutes 39 seconds. 🔴

• Link to the test results.

• Link to the reports index.

Test configuration:

# RHOAI: run fine_tuning test test_ci
PR_POSITIONAL_ARGS: ray
PR_POSITIONAL_ARG_0: fine_tuning-perf-ci
PR_POSITIONAL_ARG_1: ray

• Link to the Rebuild page.

Failure indicator:

/logs/artifacts/000_test_ci/002__test_fine_tuning/000__fine_tuning__ray_fine_tuning_job/FAILURE | [000__fine_tuning__ray_fine_tuning_job] ./run_toolbox.py from_config fine_tuning ray_fine_tuning_job --extra={'name': 'fine-tuning', 'model_name': 'bigscience/bloom-560m@hf', 'dataset_name': 'twitter_complaints_small.json', 'gpu': 1, 'dataset_replication': 1} --> 2
/logs/artifacts/000_test_ci/002__test_fine_tuning/FAILURE | CalledProcessError: Command 'set -o errexit;set -o pipefail;set -o nounset;set -o errtrace;ARTIFACT_DIR="/logs/artifacts/000_test_ci/002__test_fine_tuning" ./run_toolbox.py from_config fine_tuning ray_fine_tuning_job --extra="{'name': 'fine-tuning', 'model_name': 'bigscience/bloom-560m@hf', 'dataset_name': 'twitter_complaints_small.json', 'gpu': 1, 'dataset_replication': 1}"' returned non-zero exit status 2.
Traceback (most recent call last):
  File "/opt/topsail/src/projects/fine_tuning/testing/test_finetuning.py", line 132, in _run_test
    run.run_toolbox_from_config(
  File "/opt/topsail/src/projects/core/library/run.py", line 49, in run_toolbox_from_config
    return run(f'{cmd_env} ./run_toolbox.py from_config {group} {command} {_dict_to_run_toolbox_args(kwargs)}', **run_kwargs)
  File "/opt/topsail/src/projects/core/library/run.py", line 105, in run
    proc = subprocess.run(command, **args)
  File "/usr/lib64/python3.9/subprocess.py", line 528, in run

[...]

[Test ran on the internal Perflab CI]

topsail-bot · 2024-11-06T10:46:42Z

Jenkins Job #1608

🔴 Test of 'rhoai test test_ci' failed after 00 hours 00 minutes 06 seconds. 🔴

• Link to the test results.

• Link to the reports index.

Test configuration:

# RHOAI: run fine_tuning test test_ci
PR_POSITIONAL_ARGS: ray_bench
PR_POSITIONAL_ARG_0: fine_tuning-perf-ci
PR_POSITIONAL_ARG_1: ray_bench

• Link to the Rebuild page.

Failure indicator:

/logs/artifacts/000_test_ci/FAILURE | Traceback (most recent call last):
  File "/opt/topsail/src/projects/fine_tuning/testing/test_finetuning.py", line 368, in test
    failed = _run_test_and_visualize()
  File "/opt/topsail/src/projects/fine_tuning/testing/test_finetuning.py", line 288, in _run_test_and_visualize
    failed = _run_test(test_artifact_dir_p, test_override_values)
  File "/opt/topsail/src/projects/fine_tuning/testing/test_finetuning.py", line 87, in _run_test
    dataset_source = sources[test_settings["dataset_name"]]
KeyError: None

[Test ran on the internal Perflab CI]

kpouget · 2024-11-06T14:06:21Z

/test rhoai-light fine_tuning ray_bench

openshift-ci · 2024-11-06T15:24:00Z

@kpouget: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/rhoai-light	`47f946e`	link	true	`/test rhoai-light`

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

topsail-bot · 2024-11-06T19:03:42Z

Jenkins Job #1610

🔴 Test of 'rhoai test prepare_ci' failed after 00 hours 05 minutes 18 seconds. 🔴

• Link to the test results.

• Link to the reports index.

Test configuration:

# RHOAI: run fine_tuning test prepare_ci
PR_POSITIONAL_ARGS: ray_bench
PR_POSITIONAL_ARG_0: fine_tuning-perf-ci
PR_POSITIONAL_ARG_1: ray_bench

• Link to the Rebuild page.

Failure indicator:

/logs/artifacts/000_prepare_ci/001__prepare2/000__prepare_namespace/FAILURE | UnboundLocalError: local variable 'model' referenced before assignment
Traceback (most recent call last):
  File "/opt/topsail/src/projects/fine_tuning/testing/prepare_finetuning.py", line 224, in prepare_namespace
    download_data_sources(test_settings)
  File "/opt/topsail/src/projects/fine_tuning/testing/prepare_finetuning.py", line 113, in download_data_sources
    elif isinstance(model, list):
UnboundLocalError: local variable 'model' referenced before assignment

[Test ran on the internal Perflab CI]

topsail-bot · 2024-11-07T11:26:29Z

Jenkins Job #1615

🔴 Test of 'rhoai test test_ci' failed after 00 hours 07 minutes 47 seconds. 🔴

• Link to the test results.

• Link to the reports index.

Test configuration:

# RHOAI: run fine_tuning test test_ci
PR_POSITIONAL_ARGS: ray_bench
PR_POSITIONAL_ARG_0: fine_tuning-perf-ci
PR_POSITIONAL_ARG_1: ray_bench

• Link to the Rebuild page.

Failure indicator:

/logs/artifacts/000_test_ci/001__ray__ray-benchmark/000__fine_tuning__ray_fine_tuning_job/FAILURE | [000__fine_tuning__ray_fine_tuning_job] ./run_toolbox.py from_config fine_tuning ray_fine_tuning_job --extra={'name': 'fine-tuning', 'gpu': 1} --> 2
/logs/artifacts/000_test_ci/001__ray__ray-benchmark/FAILURE | CalledProcessError: Command 'set -o errexit;set -o pipefail;set -o nounset;set -o errtrace;ARTIFACT_DIR="/logs/artifacts/000_test_ci/001__ray__ray-benchmark" ./run_toolbox.py from_config fine_tuning ray_fine_tuning_job --extra="{'name': 'fine-tuning', 'gpu': 1}"' returned non-zero exit status 2.
Traceback (most recent call last):
  File "/opt/topsail/src/projects/fine_tuning/testing/test_finetuning.py", line 139, in _run_test
    run.run_toolbox_from_config(
  File "/opt/topsail/src/projects/core/library/run.py", line 49, in run_toolbox_from_config
    return run(f'{cmd_env} ./run_toolbox.py from_config {group} {command} {_dict_to_run_toolbox_args(kwargs)}', **run_kwargs)
  File "/opt/topsail/src/projects/core/library/run.py", line 105, in run
    proc = subprocess.run(command, **args)
  File "/usr/lib64/python3.9/subprocess.py", line 528, in run

[...]

[Test ran on the internal Perflab CI]

topsail-bot · 2024-11-07T12:58:14Z

Jenkins Job #1616

🔴 Test of 'rhoai test test_ci' failed after 00 hours 07 minutes 16 seconds. 🔴

• Link to the test results.

• Link to the reports index.

Test configuration:

# RHOAI: run fine_tuning test test_ci
PR_POSITIONAL_ARGS: ray_bench
PR_POSITIONAL_ARG_0: fine_tuning-perf-ci
PR_POSITIONAL_ARG_1: ray_bench

• Link to the Rebuild page.

Failure indicator:

/logs/artifacts/000_test_ci/001__ray__ray-benchmark/000__fine_tuning__ray_fine_tuning_job/FAILURE | [000__fine_tuning__ray_fine_tuning_job] ./run_toolbox.py from_config fine_tuning ray_fine_tuning_job --extra={'name': 'fine-tuning', 'gpu': 1} --> 2
/logs/artifacts/000_test_ci/001__ray__ray-benchmark/FAILURE | CalledProcessError: Command 'set -o errexit;set -o pipefail;set -o nounset;set -o errtrace;ARTIFACT_DIR="/logs/artifacts/000_test_ci/001__ray__ray-benchmark" ./run_toolbox.py from_config fine_tuning ray_fine_tuning_job --extra="{'name': 'fine-tuning', 'gpu': 1}"' returned non-zero exit status 2.
Traceback (most recent call last):
  File "/opt/topsail/src/projects/fine_tuning/testing/test_finetuning.py", line 139, in _run_test
    run.run_toolbox_from_config(
  File "/opt/topsail/src/projects/core/library/run.py", line 49, in run_toolbox_from_config
    return run(f'{cmd_env} ./run_toolbox.py from_config {group} {command} {_dict_to_run_toolbox_args(kwargs)}', **run_kwargs)
  File "/opt/topsail/src/projects/core/library/run.py", line 105, in run
    proc = subprocess.run(command, **args)
  File "/usr/lib64/python3.9/subprocess.py", line 528, in run

[...]

[Test ran on the internal Perflab CI]

topsail-bot · 2024-11-07T13:25:54Z

Jenkins Job #1617

🔴 Test of 'rhoai test test_ci' failed after 00 hours 07 minutes 27 seconds. 🔴

• Link to the test results.

• Link to the reports index.

Test configuration:

# RHOAI: run fine_tuning test test_ci
PR_POSITIONAL_ARGS: ray_bench
PR_POSITIONAL_ARG_0: fine_tuning-perf-ci
PR_POSITIONAL_ARG_1: ray_bench

• Link to the Rebuild page.

Failure indicator:

/logs/artifacts/000_test_ci/001__ray__ray-benchmark/000__fine_tuning__ray_fine_tuning_job/FAILURE | [000__fine_tuning__ray_fine_tuning_job] ./run_toolbox.py from_config fine_tuning ray_fine_tuning_job --extra={'name': 'fine-tuning', 'gpu': 1, 'hyper_parameters': {'num_samples': 10}} --> 2
/logs/artifacts/000_test_ci/001__ray__ray-benchmark/FAILURE | CalledProcessError: Command 'set -o errexit;set -o pipefail;set -o nounset;set -o errtrace;ARTIFACT_DIR="/logs/artifacts/000_test_ci/001__ray__ray-benchmark" ./run_toolbox.py from_config fine_tuning ray_fine_tuning_job --extra="{'name': 'fine-tuning', 'gpu': 1, 'hyper_parameters': {'num_samples': 10}}"' returned non-zero exit status 2.
Traceback (most recent call last):
  File "/opt/topsail/src/projects/fine_tuning/testing/test_finetuning.py", line 139, in _run_test
    run.run_toolbox_from_config(
  File "/opt/topsail/src/projects/core/library/run.py", line 49, in run_toolbox_from_config
    return run(f'{cmd_env} ./run_toolbox.py from_config {group} {command} {_dict_to_run_toolbox_args(kwargs)}', **run_kwargs)
  File "/opt/topsail/src/projects/core/library/run.py", line 105, in run
    proc = subprocess.run(command, **args)
  File "/usr/lib64/python3.9/subprocess.py", line 528, in run

[...]

[Test ran on the internal Perflab CI]

topsail-bot · 2024-11-07T14:27:50Z

Jenkins Job #1618

🔴 Test of 'rhoai test test_ci' failed after 00 hours 02 minutes 17 seconds. 🔴

• Link to the test results.

• Link to the reports index.

Test configuration:

# RHOAI: run fine_tuning test test_ci
PR_POSITIONAL_ARGS: ray_bench
PR_POSITIONAL_ARG_0: fine_tuning-perf-ci
PR_POSITIONAL_ARG_1: ray_bench

• Link to the Rebuild page.

Failure indicator:

/logs/artifacts/000_test_ci/001__ray__ray-benchmark/000__fine_tuning__ray_fine_tuning_job/FAILURE | [000__fine_tuning__ray_fine_tuning_job] ./run_toolbox.py from_config fine_tuning ray_fine_tuning_job --extra={'name': 'fine-tuning', 'gpu': 1, 'hyper_parameters': {'num_samples': 10}} --> 2
/logs/artifacts/000_test_ci/001__ray__ray-benchmark/FAILURE | CalledProcessError: Command 'set -o errexit;set -o pipefail;set -o nounset;set -o errtrace;ARTIFACT_DIR="/logs/artifacts/000_test_ci/001__ray__ray-benchmark" ./run_toolbox.py from_config fine_tuning ray_fine_tuning_job --extra="{'name': 'fine-tuning', 'gpu': 1, 'hyper_parameters': {'num_samples': 10}}"' returned non-zero exit status 2.
Traceback (most recent call last):
  File "/opt/topsail/src/projects/fine_tuning/testing/test_finetuning.py", line 139, in _run_test
    run.run_toolbox_from_config(
  File "/opt/topsail/src/projects/core/library/run.py", line 49, in run_toolbox_from_config
    return run(f'{cmd_env} ./run_toolbox.py from_config {group} {command} {_dict_to_run_toolbox_args(kwargs)}', **run_kwargs)
  File "/opt/topsail/src/projects/core/library/run.py", line 105, in run
    proc = subprocess.run(command, **args)
  File "/usr/lib64/python3.9/subprocess.py", line 528, in run

[...]

[Test ran on the internal Perflab CI]

topsail-bot · 2024-11-07T15:35:34Z

Jenkins Job #1620

🟢 Test of 'rhoai test test_ci' succeeded after 00 hours 06 minutes 47 seconds. 🟢

• Link to the test results.

• Link to the reports index.

Test configuration:

# RHOAI: run fine_tuning test test_ci
PR_POSITIONAL_ARGS: ray_bench
PR_POSITIONAL_ARG_0: fine_tuning-perf-ci
PR_POSITIONAL_ARG_1: ray_bench

• Link to the Rebuild page.

[Test ran on the internal Perflab CI]

topsail-bot · 2024-11-07T16:04:51Z

Jenkins Job #1621

🟢 Test of 'rhoai test test_ci' succeeded after 00 hours 07 minutes 32 seconds. 🟢

• Link to the test results.

• Link to the reports index.

Test configuration:

# RHOAI: run fine_tuning test test_ci
PR_POSITIONAL_ARGS: ''
PR_POSITIONAL_ARG_0: fine_tuning-perf-ci

• Link to the Rebuild page.

[Test ran on the internal Perflab CI]

…-job.yaml: remove unused file

…iable

…se a workload (entrypoint+app) directory

…-benchmark: new workload

…son presets

…aset/model/pvc

…-benchmark/app/test_network_overhead: allow configuring the num_samples

kpouget · 2024-11-07T16:24:12Z

tests passed ❤️ , merging

kpouget force-pushed the ray branch 4 times, most recently from 6dad7cc to 17b59f2 Compare November 5, 2024 15:14

kpouget force-pushed the ray branch from 17b59f2 to 151a3bc Compare November 5, 2024 15:33

kpouget force-pushed the ray branch from 151a3bc to b50c385 Compare November 5, 2024 20:37

kpouget force-pushed the ray branch from b50c385 to 30d4543 Compare November 5, 2024 21:05

kpouget force-pushed the ray branch 3 times, most recently from a94a24a to 5cde8ab Compare November 6, 2024 08:25

kpouget force-pushed the ray branch from 5cde8ab to 0fc7c75 Compare November 6, 2024 08:55

kpouget force-pushed the ray branch from 0fc7c75 to 1ebdfb4 Compare November 6, 2024 09:30

kpouget force-pushed the ray branch from 1ebdfb4 to 7934227 Compare November 6, 2024 10:37

kpouget force-pushed the ray branch from 7934227 to 47f946e Compare November 6, 2024 11:03

kpouget force-pushed the ray branch from 47f946e to 177d88a Compare November 7, 2024 08:04

kpouget force-pushed the ray branch from a69a987 to 5a081a5 Compare November 7, 2024 12:44

kpouget force-pushed the ray branch 3 times, most recently from 1024d7b to 35b7a15 Compare November 7, 2024 13:10

kpouget force-pushed the ray branch from 35b7a15 to da82725 Compare November 7, 2024 14:12

kpouget force-pushed the ray branch 2 times, most recently from 9e3c026 to c56a043 Compare November 7, 2024 15:14

kpouget force-pushed the ray branch from c56a043 to d0582cd Compare November 7, 2024 16:20

kpouget mentioned this pull request Nov 7, 2024

[cluster] library: prom: with-queries: load the 'metrics.txt' file from the 'testing_dir' #582

Merged

kpouget added 10 commits November 7, 2024 17:23

[fine_tuning] testing: integrate Ray in the fine-tuning orchestration

693929d

[fine_tuning] toolbox: fine_tuning_ray_fine_tuning_job: templates/ray…

71e0f6b

…-job.yaml: remove unused file

[fine_tuning] toolbox: fine_tuning_ray_fine_tuning_job: make more rel…

3535935

…iable

[fine_tuning] toolbox: fine_tuning_ray_fine_tuning_job: refactor to u…

8e2938a

…se a workload (entrypoint+app) directory

[fine_tuning] toolbox: fine_tuning_ray_fine_tuning_job: workloads/ray…

0c32af7

…-benchmark: new workload

[fine_tuning] testing: config: remove outdated IBM regression compari…

2575c26

…son presets

[core] library: ansible_toolbox: remove stray pdb call

47c61c7

[fine_tuning] testing: config: add ray_bench preset

ad0e11b

[fine_tuning] testing: prepare_finetuning: make it work without a dat…

34598f1

…aset/model/pvc

[fine_tuning] toolbox: fine_tuning_ray_fine_tuning_job: workloads/ray…

ca3ecbf

…-benchmark/app/test_network_overhead: allow configuring the num_samples

kpouget force-pushed the ray branch from d0582cd to ca3ecbf Compare November 7, 2024 16:23

kpouget enabled auto-merge November 7, 2024 16:24

kpouget merged commit a268e0b into openshift-psap:main Nov 7, 2024
6 checks passed

kpouget deleted the ray branch November 7, 2024 16:24

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[fine-tuning] Integrate Ray benchmarking as an alternative fine-tuning job #580

[fine-tuning] Integrate Ray benchmarking as an alternative fine-tuning job #580

kpouget commented Nov 5, 2024

openshift-ci bot commented Nov 5, 2024

topsail-bot bot commented Nov 5, 2024

topsail-bot bot commented Nov 5, 2024

topsail-bot bot commented Nov 5, 2024

topsail-bot bot commented Nov 5, 2024

topsail-bot bot commented Nov 5, 2024

topsail-bot bot commented Nov 5, 2024

topsail-bot bot commented Nov 5, 2024

topsail-bot bot commented Nov 6, 2024

topsail-bot bot commented Nov 6, 2024

topsail-bot bot commented Nov 6, 2024

topsail-bot bot commented Nov 6, 2024

kpouget commented Nov 6, 2024

openshift-ci bot commented Nov 6, 2024

topsail-bot bot commented Nov 6, 2024

topsail-bot bot commented Nov 7, 2024

topsail-bot bot commented Nov 7, 2024

topsail-bot bot commented Nov 7, 2024

topsail-bot bot commented Nov 7, 2024

topsail-bot bot commented Nov 7, 2024

topsail-bot bot commented Nov 7, 2024

kpouget commented Nov 7, 2024

[fine-tuning] Integrate Ray benchmarking as an alternative fine-tuning job #580

[fine-tuning] Integrate Ray benchmarking as an alternative fine-tuning job #580

Conversation

kpouget commented Nov 5, 2024

openshift-ci bot commented Nov 5, 2024

topsail-bot bot commented Nov 5, 2024

topsail-bot bot commented Nov 5, 2024

topsail-bot bot commented Nov 5, 2024

topsail-bot bot commented Nov 5, 2024

topsail-bot bot commented Nov 5, 2024

topsail-bot bot commented Nov 5, 2024

topsail-bot bot commented Nov 5, 2024

topsail-bot bot commented Nov 6, 2024

topsail-bot bot commented Nov 6, 2024

topsail-bot bot commented Nov 6, 2024

topsail-bot bot commented Nov 6, 2024

kpouget commented Nov 6, 2024

openshift-ci bot commented Nov 6, 2024

topsail-bot bot commented Nov 6, 2024

topsail-bot bot commented Nov 7, 2024

topsail-bot bot commented Nov 7, 2024

topsail-bot bot commented Nov 7, 2024

topsail-bot bot commented Nov 7, 2024

topsail-bot bot commented Nov 7, 2024

topsail-bot bot commented Nov 7, 2024

kpouget commented Nov 7, 2024