Update tests for transformers 4.36 #10858

jenniew · 2024-04-23T07:33:37Z

Update tests for transformers 4.36.2

Related issue: https://github.com/analytics-zoo/nano/issues/1289

Move mistral related test to main tests as mistral is working under transformers 4.36. Removed 4.34 tests

…to transformer4.36_tests

jenniew · 2024-05-10T03:04:47Z

Examples are also verified. https://github.com/analytics-zoo/nano/issues/1120

liu-shaojun · 2024-05-11T02:20:57Z

.github/workflows/llm_performance_tests.yml

  workflow_dispatch:
  workflow_call:

 # A workflow run is made up of one or more jobs that can run sequentially or in parallel
 jobs:
-  # llm-cpp-build: # please uncomment it for PR tests
-  #   uses: ./.github/workflows/llm-binary-build.yml
+  llm-cpp-build: # please uncomment it for PR tests


Please revert these changes before PR merge

There seems to be other places changed for PR tests that should be reverted

hkvision · 2024-05-14T06:08:50Z

python/llm/test/benchmark/arc-perf-test.yaml

  - 'Qwen/Qwen-7B-Chat'
  - 'BAAI/AquilaChat-7B'
  - 'baichuan-inc/Baichuan2-7B-Chat'
  - 'baichuan-inc/Baichuan2-13B-Chat-4bit'
  - 'bigscience/bloomz-7b1'
-  - 'fnlp/moss-moon-003-sft-4bit'
+#  - 'fnlp/moss-moon-003-sft-4bit' # moss-moon-003-sft cannot work on transformers 4.34+


is this supposed to be fixed?

This is the tokenizer issue of moss-moon-003-sft model, because moss-moon-003-sft haven't fix the tokenizer issue to compatible with transformers 4.34+, we can not fix it. See the issue: https://github.com/analytics-zoo/nano/issues/1145

Then shall we keep transformers 4.31 in the test as well for this model?

I don't know whether we need to keep such test for transformers 4.31 since ipex-llm would be updated to support transformers 4.36. @jason-dai Do we need to keep tests which only works on transformers 4.31?

hkvision · 2024-05-14T06:13:31Z

.github/workflows/llm-whisper-evaluation.yml

@@ -75,7 +75,7 @@ jobs:
            echo "runner=$runner" >> $GITHUB_OUTPUT

  llm-whisper-evaluation:
-    # if: ${{ github.event.schedule || github.event.inputs.artifact == 'llm-whisper-evaluation' || github.event.inputs.artifact == 'all' }} # please comment it for PR tests


revert this change to make this PR more clean.

hkvision · 2024-05-14T06:14:00Z

http://10.239.44.136:8888/pr_perf_gpu/ Need to check the performance before the merge.

Oscilloscope98 · 2024-05-14T06:18:22Z

.github/workflows/llm_performance_tests.yml

  workflow_dispatch:
  workflow_call:

 # A workflow run is made up of one or more jobs that can run sequentially or in parallel
 jobs:
-  # llm-cpp-build: # please uncomment it for PR tests
-  #   uses: ./.github/workflows/llm-binary-build.yml
+  llm-cpp-build: # please uncomment it for PR tests


There seems to be other places changed for PR tests that should be reverted

.github/workflows/llm_performance_tests.yml

liu-shaojun · 2024-05-14T06:16:09Z

python/llm/test/run-llm-inference-tests.sh

@@ -18,10 +18,6 @@ export OMP_NUM_THREADS=$THREAD_NUM
 python -m pytest -s ${LLM_INFERENCE_TEST_DIR}/test_transformers_api.py -v
 python -m pytest -s ${LLM_INFERENCE_TEST_DIR}/test_optimize_model_api.py -v

-python -m pip install transformers==4.34.0


Does the testing for Transformers 4.34 is no longer necessary?

should be, mistral is put in transformers 4.36, which replaces 4.34

yes, the testing for Transformers 4.34 is no longer necessary, since I move mistral test into general test cases.

liu-shaojun · 2024-05-14T07:34:28Z

.github/workflows/llm_performance_tests.yml


  llm-performance-test-on-arc:
-    if: ${{ github.event.schedule || github.event_name == 'workflow_dispatch' || github.event.inputs.artifact == 'llm-performance-test-on-arc' || github.event.inputs.artifact == 'all' }} # please comment it for PR tests
-    # needs: llm-cpp-build # please uncomment it for PR tests
+    #if: ${{ github.event.schedule || github.event_name == 'workflow_dispatch' || github.event.inputs.artifact == 'llm-performance-test-on-arc' || github.event.inputs.artifact == 'all' }} # please comment it for PR tests


For arc/core/spr perf test, except for the code that needs to be reverted, other changes LGTM. For igpu test, please refer to yuwen’s comment.

liu-shaojun · 2024-05-14T07:35:59Z

python/llm/test/inference/test_transformers_api.py

        self.assertTrue(res)

    def test_transformers_auto_model_for_causal_lm_int4(self):
-        model_path = os.environ.get('ORIGINAL_REPLIT_CODE_PATH')
+        model_path = os.environ.get('ORIGINAL_CODESHELL_7B_PATH')


What is the reason for this change?

replit-code-v1-3b cannot run with transformers 4.36, so another code generation model is used here instead :)

liu-shaojun · 2024-05-14T07:37:37Z

.github/workflows/llm_unit_tests.yml

@@ -99,7 +99,7 @@ jobs:
          echo "LLAMA_ORIGIN_PATH=${ORIGIN_DIR}/llama-7b-hf" >> "$GITHUB_ENV"
          echo "BLOOM_ORIGIN_PATH=${ORIGIN_DIR}/bloom-7b1" >> "$GITHUB_ENV"
          echo "ORIGINAL_CHATGLM2_6B_PATH=${ORIGIN_DIR}/chatglm2-6b" >> "$GITHUB_ENV"
-          echo "ORIGINAL_REPLIT_CODE_PATH=${ORIGIN_DIR}/replit-code-v1-3b" >> "$GITHUB_ENV"
+          echo "ORIGINAL_CODESHELL_7B_PATH=${ORIGIN_DIR}/CodeShell-7B-Chat" >> "$GITHUB_ENV"


What is the reason for this change?

replit-code-v1-3b cannot run with transformers 4.36, so another code generation model is used here instead :)

…to transformer4.36_tests

jenniew · 2024-05-22T02:33:43Z

http://10.239.44.136:8888/pr_perf_gpu/ Need to check the performance before the merge.

@hkvision @lalalapotter According to the performance report, seems mpt-7b-chat has some performance gap.
For mpt-7b-chat, we upgrade the model since the old model cannot be compatible with transformers 4.36, but the new model implementation has been changed. So we may need to update our optimization for the new model.

hkvision

The Arc related changes LGTM.

Oscilloscope98 · 2024-05-23T10:19:52Z

.github/workflows/llm_performance_tests.yml

+    paths:
+      - ".github/workflows/llm_performance_tests.yml"
+      - "python/llm/test/benchmark/**"
+      - "python/llm/dev/benchmark/all-in-one/**"


We should revert the change in this file which was for PR tests; any only leave necessary changes :)

liu-shaojun

LGTM

jenniew added 2 commits April 23, 2024 00:30

update unit test

f9d0107

update

a86c35f

jenniew requested review from glorysdj and liu-shaojun as code owners April 23, 2024 07:33

jenniew added 26 commits April 23, 2024 00:38

update

8a6e0f2

update

d658968

update

66639dc

update

e77cee4

fix gpu attention test

f1d6944

update

c2fa88b

update

b255ac5

update

a82199a

update

7e7d09c

update

8f1c355

Merge branch 'main' of https://github.com/intel-analytics/ipex-llm in…

0e7f73a

…to transformer4.36_tests

update

e0c4407

update example test

c51b7ea

replace replit code

a442768

update

5563f28

update

b575c48

update

cc0ed30

update

04333ae

set safe_serialization false

8ecdeac

perf test

49a6933

merge

e52180c

update

9217662

update

3ad25b7

update

8ee92d2

update

45d2383

update

e968252

jenniew added 5 commits May 8, 2024 11:58

delete

a533ae8

update

4af1445

update

1f91353

update

0696491

update

6922dc7

jenniew requested review from Oscilloscope98 and hkvision May 10, 2024 02:48

liu-shaojun reviewed May 11, 2024

View reviewed changes

hkvision reviewed May 14, 2024

View reviewed changes

Oscilloscope98 reviewed May 14, 2024

View reviewed changes

liu-shaojun reviewed May 14, 2024

View reviewed changes

jenniew added 5 commits May 14, 2024 14:33

update

6417726

merge

e30a397

update

ec2cd5e

merge

bc1fec0

Merge branch 'main' of https://github.com/intel-analytics/ipex-llm in…

55fee3b

…to transformer4.36_tests

hkvision approved these changes May 23, 2024

View reviewed changes

Oscilloscope98 reviewed May 23, 2024

View reviewed changes

jenniew added 2 commits May 23, 2024 16:14

revert

dcd8115

update

936fafe

liu-shaojun approved these changes May 24, 2024

View reviewed changes

Oscilloscope98 approved these changes May 24, 2024

View reviewed changes

hkvision merged commit 0a06a6e into intel-analytics:main May 24, 2024
50 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update tests for transformers 4.36 #10858

Update tests for transformers 4.36 #10858

jenniew commented Apr 23, 2024 •

edited

Loading

jenniew commented May 10, 2024

liu-shaojun May 11, 2024

jenniew May 13, 2024

Oscilloscope98 May 14, 2024 •

edited

Loading

jenniew May 24, 2024

hkvision May 14, 2024

jenniew May 14, 2024

hkvision May 23, 2024

jenniew May 23, 2024

hkvision May 14, 2024

jenniew May 24, 2024

hkvision commented May 14, 2024

Oscilloscope98 May 14, 2024 •

edited

Loading

liu-shaojun May 14, 2024

hkvision May 14, 2024

jenniew May 14, 2024

liu-shaojun May 14, 2024

liu-shaojun May 14, 2024

jenniew May 14, 2024

liu-shaojun May 14, 2024

Oscilloscope98 May 14, 2024

jenniew commented May 22, 2024 •

edited

Loading

hkvision left a comment •

edited

Loading

Oscilloscope98 May 23, 2024 •

edited

Loading

jenniew May 24, 2024

liu-shaojun left a comment

Update tests for transformers 4.36 #10858

Update tests for transformers 4.36 #10858

Conversation

jenniew commented Apr 23, 2024 • edited Loading

jenniew commented May 10, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Oscilloscope98 May 14, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hkvision commented May 14, 2024

Oscilloscope98 May 14, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jenniew commented May 22, 2024 • edited Loading

hkvision left a comment • edited Loading

Choose a reason for hiding this comment

Oscilloscope98 May 23, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

liu-shaojun left a comment

Choose a reason for hiding this comment

jenniew commented Apr 23, 2024 •

edited

Loading

Oscilloscope98 May 14, 2024 •

edited

Loading

Oscilloscope98 May 14, 2024 •

edited

Loading

jenniew commented May 22, 2024 •

edited

Loading

hkvision left a comment •

edited

Loading

Oscilloscope98 May 23, 2024 •

edited

Loading