Error: draw graphs from json

[anandhu-eng]

---

Migrated from mlcommons/mlperf-automations_archived#49
Originally created by @anandhu-eng on Thu, 05 Dec 2024 14:23:47 GMT

---

DEBUG:root:  - Running native script "/home/cmuser/CM/repos/mlcommons@cm4mlops/script/draw-graph-from-json-data/run.sh" from temporal script "tmp-run.sh" in "/cm-mount/home/anandhu/CM/repos/mlcommons@cm4mlops/test_results/8302dcf023dd-reference-gpu-pytorch-v2.1.0-cu124/rgat/offline/performance/run_1" ...
INFO:root:       ! cd /cm-mount/home/anandhu/CM/repos/mlcommons@cm4mlops/test_results/8302dcf023dd-reference-gpu-pytorch-v2.1.0-cu124/rgat/offline/performance/run_1
INFO:root:       ! call /home/cmuser/CM/repos/mlcommons@cm4mlops/script/draw-graph-from-json-data/run.sh from tmp-run.sh
Running: 
/home/cmuser/venv/cm/bin/python3 /home/cmuser/CM/repos/mlcommons@cm4mlops/script/draw-graph-from-json-data/process-cm-deps.py  /cm-mount/home/anandhu/CM/repos/mlcommons@cm4mlops/test_results/8302dcf023dd-reference-gpu-pytorch-v2.1.0-cu124/rgat/offline/performance/run_1/cm-version-info.json --output_image /cm-mount/home/anandhu/CM/repos/mlcommons@cm4mlops/test_results/8302dcf023dd-reference-gpu-pytorch-v2.1.0-cu124/rgat/offline/performance/run_1/cm-deps.png --output_mermaid /cm-mount/home/anandhu/CM/repos/mlcommons@cm4mlops/test_results/8302dcf023dd-reference-gpu-pytorch-v2.1.0-cu124/rgat/offline/performance/run_1/cm-deps.mmd

Traceback (most recent call last):
  File "/home/cmuser/CM/repos/mlcommons@cm4mlops/script/draw-graph-from-json-data/process-cm-deps.py", line 122, in <module>
    main()
  File "/home/cmuser/CM/repos/mlcommons@cm4mlops/script/draw-graph-from-json-data/process-cm-deps.py", line 112, in main
    G = generate_graph_from_nested_json(
  File "/home/cmuser/CM/repos/mlcommons@cm4mlops/script/draw-graph-from-json-data/process-cm-deps.py", line 66, in generate_graph_from_nested_json
    plt.savefig(output_image, format="png", dpi=300)
  File "/home/cmuser/venv/cm/lib/python3.10/site-packages/matplotlib/pyplot.py", line 1228, in savefig
    res = fig.savefig(*args, **kwargs)  # type: ignore[func-returns-value]
  File "/home/cmuser/venv/cm/lib/python3.10/site-packages/matplotlib/figure.py", line 3395, in savefig
    self.canvas.print_figure(fname, **kwargs)
  File "/home/cmuser/venv/cm/lib/python3.10/site-packages/matplotlib/backend_bases.py", line 2204, in print_figure
    result = print_method(
  File "/home/cmuser/venv/cm/lib/python3.10/site-packages/matplotlib/backend_bases.py", line 2054, in <lambda>
    print_method = functools.wraps(meth)(lambda *args, **kwargs: meth(
  File "/home/cmuser/venv/cm/lib/python3.10/site-packages/matplotlib/backends/backend_agg.py", line 496, in print_png
    self._print_pil(filename_or_obj, "png", pil_kwargs, metadata)
  File "/home/cmuser/venv/cm/lib/python3.10/site-packages/matplotlib/backends/backend_agg.py", line 444, in _print_pil
    FigureCanvasAgg.draw(self)
  File "/home/cmuser/venv/cm/lib/python3.10/site-packages/matplotlib/backends/backend_agg.py", line 387, in draw
    self.figure.draw(self.renderer)
  File "/home/cmuser/venv/cm/lib/python3.10/site-packages/matplotlib/artist.py", line 95, in draw_wrapper
    result = draw(artist, renderer, *args, **kwargs)
  File "/home/cmuser/venv/cm/lib/python3.10/site-packages/matplotlib/artist.py", line 72, in draw_wrapper
    return draw(artist, renderer)
  File "/home/cmuser/venv/cm/lib/python3.10/site-packages/matplotlib/figure.py", line 3161, in draw
    self.patch.draw(renderer)
  File "/home/cmuser/venv/cm/lib/python3.10/site-packages/matplotlib/artist.py", line 72, in draw_wrapper
    return draw(artist, renderer)
  File "/home/cmuser/venv/cm/lib/python3.10/site-packages/matplotlib/patches.py", line 632, in draw
    self._draw_paths_with_artist_properties(
  File "/home/cmuser/venv/cm/lib/python3.10/site-packages/matplotlib/patches.py", line 617, in _draw_paths_with_artist_properties
    renderer.draw_path(gc, *draw_path_args)
  File "/home/cmuser/venv/cm/lib/python3.10/site-packages/matplotlib/backends/backend_agg.py", line 131, in draw_path
    self._renderer.draw_path(gc, path, transform, rgbFace)
ValueError: object __array__ method not producing an array

CM error: Portable CM script failed (name = draw-graph-from-json-data, return code = 256)

[arjunsuresh]

Is this issue currently existing? Our nightly runs are producing the graphs without any issue.

[anandhu-eng]

Hi @arjunsuresh this issue was produced in the server machine. Maybe there is a requirement for specific numpy version?

[anandhu-eng]

I could recheck during the llama3.1 test run with the 8 billion model.