tflite-runtime (2.12 and newer) no longer supports EdgeTPU

[feranick]

Description

While tflite-runtime can be successfully compile from source (using the full TensorFlow source) the whl produced does not have support for EdgeTPU (it's not recognized within tflite-runtime). In the past coral.ai used to provide custom-made tflite-runtime with EdgeTPU support, yet, those are either no longer available, or they are built again obsolete versions of TF/Python. Furthermore, while coral.ai page indicate that such whl can be found in here, that page refers back to the original coral.ai page for support for EdgeTPU. This endless loop is confusing and pointless. The version on Pypi doesn't have support for EdgeTPU either. One may suggest the use of PyCoral. Well PyCoral uses EdgeTPU, and without it it just doesn't work.

The last version of TensorFlow Lite to support EdgeTPU is 2.11.1. Any later version leads to a segmentation fault.
Which leaves me with no way to run a model on the EdgeTPU with a recent release of TF (> 2.11.x). I tested every release since then (currently up to 2.15.0) and none works.

it would be great if Google would acknowledge that development (or at least maintenance) of TFlite for EdgeTPU is still ongoing, or if its dead.

Click to expand!

Issue Type

Build/Install, Support, Documentation Feature Request, Documentation Bug

Operating System

Linux, Ubuntu, Mac OS

Coral Device

USB Accelerator, M.2 Accelerator A+E, M.2 Accelerator B+M

Other Devices

Raspberry Pi 3

Programming Language

Python 3.8, Python 3.9, Other

Relevant Log Output

No response

[Alan01252]

My experience if it helps is 2.5.0 can be packed/compiled for later python versions.

The later versions of the runtime all fail with segfaults at runtime when trying to load the library file.

[feranick]

Thanks. I was able to get it to work with the EdgeTPU by compiling version TFlite from TF 2.11.1. Any later version of TF lead leads to segfaults.

Sounds to me Google gave up on developing EdgeTPU.

[Alan01252]

Oh that's awesome. I'll try bumping to that version too ty!!

[Skillnoob]

@feranick do you have a example on how you got it working?
I've tried with many different tflite_runtime versions and tensorflow versions and it continues to segfault.
I've posted a issue here: #821 where there's more detail.

[feranick]

@feranick do you have a example on how you got it working? I've tried with many different tflite_runtime versions and tensorflow versions and it continues to segfault. I've posted a issue here: #821 where there's more detail.

I don't have something simple I can share. However I use it within a larger software that I wrote:

https://github.com/feranick/SpectralMachine/blob/master/SpectraKeras/libSpectraKeras.py

Here's the wheel for the version that works for me (Ubuntu 22.04):

https://www.dropbox.com/scl/fi/uyif5p9g1pgn85e4u80wo/tflite_runtime-2.11.1-cp310-cp310-linux_x86_64.whl?rlkey=r7am6q6k854k0snqatde14wju&dl=0

or for MacOS:

https://www.dropbox.com/scl/fi/wikg9kdq1kij623dil66t/tflite_runtime-2.11.1-cp311-cp311-macosx_14_0_x86_64.whl?rlkey=u4e899j3tjqv15p8edr5eftfx&dl=0

[Skillnoob]

@feranick Huge thanks but sadly since I'm on a raspi 5 the wheels won't work.
Did you use this guide to make the wheel?

[feranick]

@feranick Huge thanks but sadly since I'm on a raspi 5 the wheels won't work. Did you use this guide to make the wheel?

Yes, pretty much to the letter.

[Skillnoob]

@feranick alright thanks il make my own wheel from that then.

[feranick]

@Skillnoob No problem. Please note that I followed the native compilation, not cross compilation via docker. You may try both on the RPi5. I may give it a go myself on an old RPi3 (which is all the RPI I got...).

[Skillnoob]

@feranick I'm gonna first try native and if that doesn't work cross compilation. but that will be in a few days as i am extremely busy

[feranick]

@Skillnoob I tried to compile with docker. Unfortunately the scripts are outdated as they pull outdated versions of ubuntu with python3.8. Even changing and tweaking such script leads to what looks more like a hack job than else, and compilation still doesn't succeed. I am running a native compilation with my RPi4, which is taking its sweet time, but so far hasn't failed.

FYI.

[feranick]

To be noted: Upgrading to a newer kernel 6.5 makes the edgetpu completely unusable as it breads the gasket-dkms. Even older versions of tflite-runtime will fail.

#820

[Skillnoob]

@feranick that explains the even older tflite_runtime versions not working.
I'm gonna try this next Monday or Tuesday.
Thanks a lot so far.

[feranick]

Actually, gasket-dkms is open-source and support for kernel 6.4+ has been added. I can confirm that a very quick build and reboot restores functionality of the edgetpu (with tflite_runtime <=2.11.1).

https://github.com/google/gasket-driver

[feranick]

I'd note that gasket-dkms is only needed for the edgetpu in the M2 format, which is the one I am using (not with the USB-accelerator). I have not tried the USB accelerator on linux in recent times.

[feranick]

A bit of a breakthrough. Not only gasket-dkms, but also is libedgetpu, which is what is used to compile the edgetpu drivers. The liibedgetpu drivers repo is linked directly to TF, and the version that is linked against is TF2.5.

So I forked both gasket-dkms and libedgetpu and added support for newer versions of TF, at the moment up to TF2.13.1. After compiling and installing gasket-dkms (needed only for the m2-based edgeTPU boards) and libedgetpu, edgeTPU works perfectly with tflite_runtime 2.13.1, 2.14.1, and 2.15.0. No more segmentation faults.

Currently libedgetpu does not compile with TF 2.14.0 and newer due to some python issues, but I am on working on.

All this is to say that edgetpu is back and working fine with support for modern platforms.

NOTE: the recommended way to compile libedgeetpu is via Bazel, which needs to be in the same version as TF is compiled against.

Currently tested and working with Ubuntu 22.04 x86_64 and MacOS 14.3.

[feranick]

Builds for aarch64 and armv7a are successful using my llibedgetpu fork. They are compiled with Docker+Bazel. deb packages will be available in the next day or so. Refer to updated README.md there for build instructions.

[feranick]

Builds for aarch64 and armv7a are successful using my llibedgetpu fork. They are compiled with Docker+Bazel.

Deb packages (amd64, arm64, armv7a) are available. Refer to updated README.md there for build instructions.

[Alan01252]

Amazing, thank you!

[Skillnoob]

@feranick amazing thanks for the builds.
I've run into a problem when trying to run the models.
The lib seems to rely on libc6 version 2.32 but the latest available on my raspberry pi is 2.31 and i am unable to install newer versions as that would break many other packages.

pi@raspberrypi:~/Desktop/tflite/python/examples/detection $ python3 detect_image.py   --model models/ssd_mobilenet_v2_coco_quant_postprocess_edgetpu.tflite   --labels models/coco_labels.txt   --input images/grace_hopper.bmp   --output images/grace_hopper_processed.bmp
Traceback (most recent call last):
  File "/home/pi/Desktop/tflite/python/examples/detection/detect_image.py", line 129, in <module>
    main()
  File "/home/pi/Desktop/tflite/python/examples/detection/detect_image.py", line 94, in main
    interpreter = make_interpreter(args.model)
  File "/home/pi/Desktop/tflite/python/examples/detection/detect_image.py", line 60, in make_interpreter
    tflite.load_delegate(EDGETPU_SHARED_LIB,
  File "/home/pi/.local/lib/python3.9/site-packages/tflite_runtime/interpreter.py", line 166, in load_delegate
    delegate = Delegate(library, options)
  File "/home/pi/.local/lib/python3.9/site-packages/tflite_runtime/interpreter.py", line 73, in __init__
    self._library = ctypes.pydll.LoadLibrary(library)
  File "/usr/lib/python3.9/ctypes/__init__.py", line 452, in LoadLibrary
    return self._dlltype(name)
  File "/usr/lib/python3.9/ctypes/__init__.py", line 374, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: /lib/aarch64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by /lib/aarch64-linux-gnu/libedgetpu.so.1)
Exception ignored in: <function Delegate.__del__ at 0x7f7f320700>
Traceback (most recent call last):
  File "/home/pi/.local/lib/python3.9/site-packages/tflite_runtime/interpreter.py", line 109, in __del__
    if self._library is not None:
AttributeError: 'Delegate' object has no attribute '_library'

[feranick]

I just became aware of this issue. This is because I cross-compiled from Ubuntu (with the newer library). I am setting a VM with debian at the moment...

[feranick]

OK, I reuploaded debs that should work. Raspbian is still based on bullseye, not on bookworm, hence the libc6 issue.

Please let me know if you run into issues.

[Skillnoob]

@feranick it FINALLY works. had to reinstall the os because i somehow broke my lib6c installation lmao. HUGE thanks for all the effort you put in!

[Skillnoob]

Done some testing with the pi 5 and arm64. Works well and even works on python 3.11.2 which ships with raspberry pi os bookworm (tflite_runtime 2.14.0).

[feranick]

Thanks. New builds with an updated version of libusb.

https://github.com/feranick/libedgetpu/releases/tag/v16.0TF2.13.1-2

[Skillnoob]

The updated lib's work fine on a pi 5

[joelself]

I'm going to try to get this all up and running (minus PyCoral API) tomorrow and I noticed that @feranick recently did some work to get USB working on windows. I bought the USB version because I thought it would make things easier, but the state this thing is in it looks like USB was the wrong choice. Does USB work already in Linux because that was what I was going to do everything in, but if it only works in Windows I'll be forced to do it in in Windows which is always confusing and awkward because it requires WSL or Git Bash or some Linux stuff and then things can be going fine and then get real confusing. If USB works in Linux then I can just do everything in Linux.

[feranick]

Based on my testing, the USB version works perfectly with Linux or MacOS. I have not tried Windows as I am not a user of it, but simply updated the libusb library. Frankly, I didn't even provide actual binaries for Windows (again, I don't use it)...

[joelself]

That's fine, I would like to do everything in Linux anyways because it simplifies things. Thanks for all of the work.

I was just reading some documentation and it says about PyCoral API: " A Python library with the same functionality as libcoral. (You can run inference in Python without this library, using only the tflite_runtime API.)"

So if they have the same functionality and the same language what's the point?

A bit off topic, but is there any other devboard out there that has this sort of inference power (4 TOPS in a chip, sometimes 2 chips on a board), for this kind of price? I looked everywhere and I couldn't find anything in the $20 - $150 range that is buyable. I looked everywhere and couldn't find anything. If I end up not being able to run this one I just bought, is there a tool that can emulate its performance to see if it would be good enough? It would actually suck if it worked because then I would at a point where I have a great idea for a product that would be very well priced, but I simply can't build it because all the accelerators out there are built-in to phones or come in 1U racks for $5000.

[feranick]

NOTE: as explained earlier, PyCoral is currently not supported for the latest version of TF. It may be once TF 2.17 is released. The same goes for libcoral. What is currently supported are tflite_runtime APIs.

libcoral is the C++ library. PyCoral is its wrapper for Python.

PyCoral provides a higher level API that does not rely on the TFlite APIs. So, in principle, you can access the EdgeTPU with PyCoral with very simple code as in the documentation. If however, you implemented your code and models using TF and TFlite, tflite-runtime is supported on the edgeTPU, as long as the latest libedgetpu library installed. It's pretty much your choice on how you want interact with the EdgeTPU. Personally, I prefer to do it via tflite_runtime, as my models are designed to work not just on the edgeTPU, and I don't want to rewrite code JUST for the EdgeTPU. But that is more cumbersome than PyCoral.

[joelself]

Cool, thanks for taking the time to explain it to me. It's been a while since I stuck my nose in the ANN world and back then I was fooling around with Torch instead of TF.