Kaleido 0.4 image export fails

[johannesloibl]

Hey,

just noticed some fails in my CICD that started occurring with the recent 0.4 version.
<0.4 works fine.

This is what i'm getting as error:

File "/home/runner/work/pharaoh-dev/pharaoh-dev/tests/unit/_example_assets/holoviews_plot.py", line 10, in <module>
    hv.save(model, filename="heatmap_holo_plotly.html")
  File "/home/runner/.local/share/hatch/env/virtual/pharaoh-report/3dSDsPqB/all.py3.10/lib/python3.10/site-packages/pharaoh/assetlib/patches/_holoviews.py", line 84, in patched_hv_save
    plotly_fig.write_html(filename, **kwargs)
  File "/home/runner/.local/share/hatch/env/virtual/pharaoh-report/3dSDsPqB/all.py3.10/lib/python3.10/site-packages/plotly/basedatatypes.py", line 3714, in write_html
    return pio.write_html(self, *args, **kwargs)
  File "/home/runner/.local/share/hatch/env/virtual/pharaoh-report/3dSDsPqB/all.py3.10/lib/python3.10/site-packages/pharaoh/assetlib/patches/_plotly.py", line 156, in patched_px_write_html
    patched_px_write_image(**_kwargs)
  File "/home/runner/.local/share/hatch/env/virtual/pharaoh-report/3dSDsPqB/all.py3.10/lib/python3.10/site-packages/pharaoh/assetlib/patches/_plotly.py", line 127, in patched_px_write_image
    vanilla_px_write_image(**params)
  File "/home/runner/.local/share/hatch/env/virtual/pharaoh-report/3dSDsPqB/all.py3.10/lib/python3.10/site-packages/plotly/io/_kaleido.py", line 266, in write_image
    img_data = to_image(
  File "/home/runner/.local/share/hatch/env/virtual/pharaoh-report/3dSDsPqB/all.py3.10/lib/python3.10/site-packages/plotly/io/_kaleido.py", line 143, in to_image
    img_bytes = scope.transform(
  File "/home/runner/.local/share/hatch/env/virtual/pharaoh-report/3dSDsPqB/all.py3.10/lib/python3.10/site-packages/kaleido/scopes/plotly.py", line 233, in transform
    img = kaleido.to_image_block(spec, Path(self._tempfile.name).absolute(), self._topojson, self._mapbox_access_token, debug=debug, tmp_path=self._tmp_path)
  File "/home/runner/.local/share/hatch/env/virtual/pharaoh-report/3dSDsPqB/all.py3.10/lib/python3.10/site-packages/kaleido/__init__.py", line 43, in to_image_block
    return asyncio.run(to_image(spec, f, topojson, mapbox_token, debug=debug, tmp_path=tmp_path))
  File "/opt/hostedtoolcache/Python/3.10.15/x64/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/opt/hostedtoolcache/Python/3.10.15/x64/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
    return future.result()
  File "/home/runner/.local/share/hatch/env/virtual/pharaoh-report/3dSDsPqB/all.py3.10/lib/python3.10/site-packages/kaleido/__init__.py", line 129, in to_image
    response = await tab.send_command("Runtime.callFunctionOn", params=params)
choreographer.browser.BrowserClosedError: Command not completed because browser closed.

Python:
Fails for 3.9 to 3.12

System:

Current runner version: '2.320.0'
Operating System
  Ubuntu
  [2](https://github.com/Infineon/pharaoh-dev/actions/runs/11858025206/job/33048248018#step:1:2)2.04.5
  LTS
Runner Image
  Image: ubuntu-22.04
  Version: 20241112.1.0
  Included Software: https://github.com/actions/runner-images/blob/ubuntu22/20241112.1/images/ubuntu/Ubuntu2204-Readme.md
  Image Release: https://github.com/actions/runner-images/releases/tag/ubuntu22%2F20241112.1
Runner Image Provisioner
  2.0.[3](https://github.com/Infineon/pharaoh-dev/actions/runs/11858025206/job/33048248018#step:1:3)84.1

I also have a Gitlab CI running, where this error also occurs but in a different form (also started with 0.4, so i assume it's related):

from plotly.io import to_json, to_image, write_image, write_html
  File "<frozen importlib._bootstrap>", line 1075, in _handle_fromlist
  File "/builds/global_projects/.../.hatch_data/env/virtual/all.py3.10/lib/python3.10/site-packages/_plotly_utils/importers.py", line 36, in __getattr__
    class_module = importlib.import_module(rel_module, parent_name)
  File "/usr/local/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "/builds/global_projects/.../.hatch_data/env/virtual/all.py3.10/lib/python3.10/site-packages/plotly/io/_kaleido.py", line 10, in <module>
    scope = PlotlyScope()
  File "/builds/global_projects/.../.hatch_data/env/virtual/all.py3.10/lib/python3.10/site-packages/kaleido/scopes/plotly.py", line 64, in __init__
    elif "snap" in path:
TypeError: argument of type 'NoneType' is not iterable

[ryan-williams]

I also have a daily GHA failing this AM, due to picking up Kaleido 0.4.1. I've "fixed" it by pinning to 0.2.1.

File /opt/hostedtoolcache/Python/3.11.8/x64/lib/python3.11/site-packages/plotly/basedatatypes.py:3835, in BaseFigure.write_image(self, *args, **kwargs)
   3775 """
   3776 Convert a figure to a static image and write it to a file or writeable
   3777 object
   (...)
   3831 None
   3832 """
   3833 import plotly.io as pio
-> 3835 return pio.write_image(self, *args, **kwargs)

File /opt/hostedtoolcache/Python/3.11.8/x64/lib/python3.11/site-packages/plotly/io/_kaleido.py:296, in write_image(fig, file, format, scale, width, height, validate, engine)
    286         raise ValueError(
    287             """
    288 The 'file' argument '{file}' is not a string, pathlib.Path object, or file descriptor.
   (...)
    291             )
    292         )
    293     else:
    294         # We previously succeeded in interpreting `file` as a pathlib object.
    295         # Now we can use `write_bytes()`.
--> 296         path.write_bytes(img_data)

File /opt/hostedtoolcache/Python/3.11.8/x64/lib/python3.11/pathlib.py:[106](https://github.com/hudcostreets/nj-crashes/actions/runs/11858949522/job/33050751493#step:19:107)6, in Path.write_bytes(self, data)
   1062 """
   1063 Open the file in bytes mode, write to it, and close the file.
   1064 """
   1065 # type-check for the buffer interface before truncating the file
-> 1066 view = memoryview(data)
   1067 with self.open(mode='wb') as f:
   1068     return f.write(view)

TypeError: memoryview: a bytes-like object is required, not 'NoneType'

Apologies if this should be its own issue, the error msg looks slightly similar to the last one above (something is None where it shouldn't be).

[quentin-ag]

I also faced

TypeError: argument of type 'NoneType' is not iterable

in GitLab CI with versions 0.4.0 and 0.4.1. However, 0.4.0rc5 does not have this problem.

With 0.4.0rc1 to 0.4.0rc5 I get

choreographer.browser.BrowserFailedError: Could not find an acceptable browser. Please set environmental variable BROWSER_PATH or pass `path=/path/to/browser` into the Browser() constructor.

I had to downgrade to 0.2.1 to avoid that last error.

[ayjayt]

A lot going on here!

For the first error, choreographer.browser.BrowserClosedError: Command not completed because browser closed., please set the env var KALEIDO_DEBUG to true- it will create a lot of output.

Kaleido now requires that chrome is installed separately.

I'm going to push a fix for the second error- but basically that error is that there is no chrome installed, although it shouldn't present like that.

[johannesloibl]

I also fixed it by pinning, like a already do for a long time for Windows (see #110):

"kaleido==0.1.0.post1; platform_system=='Windows'",
"kaleido<0.4; platform_system=='Linux'",

[ayjayt]

@johannesloibl @quentin-ag

Hi everyone!

If you're comfortable pinning a lower version of kaleido, that's fine- but keep in mind that chromium came with the older version and it is an older and less secure version of the browser. The browser is no longer shipped with the package (its too large), and you have to install it separately to use the newer versions of kaleido.

tl;dr: please install chrome/ium separately

[pdmkdz]

This is showing up when installing Kaleido on Job Run in databricks without set version and simply doing fig.show() in plotly.
Thanks for raising this issue, I could now ever figure out without it.

[vladsavelyev]

I thought the whole point of kaleido was is that it doesn't need external dependencies, and simplifies image exports by embedding a headless browser binary. What changed? Is there a new announcement or release notes? And is there a way to keep my Python project independent of external dependencies if it relies on Plotly image export?

[ayjayt]

I thought the whole point of kaleido was is that it doesn't need external dependencies, and simplifies image exports by embedding a headless browser binary. What changed? Is there a new announcement or release notes? And is there a way to keep my Python project independent of external dependencies if it relies on Plotly image export?

The issue is that chromium was always built into kaleido, but its gotten too large to include in the package by pypi rules- although I think we'll end up talking with them to see if they'll allow it. The other option is to include an init() function or something which does the download. I wish we had communicated better the dependency changes to avoid a lot of these issues and we're strategizing ways to make it easier for people. We may move this version of kaleido back to -rc status so that its not automatically downloaded, and we're looking at ways to repack chromium. My recommendation at this point if its too hard to download the dependency is to pin the older version, and we'll try to figure out how to wrap up that dependency internally. Sorry for the inconvenience, I really appreciate your patience.

[vladsavelyev]

Thanks so much for detailed explanation!