Runtime error while 'lv.load_layout' using klayout.lay.LayoutView()

[ooksang]

I followed your code.
I downloaded wheel file from qtless-canvas2 brach action #148.
and installed 'klayout-0.28-cp39-cp39-maylinux_2_17_x86_64.manylinux2014_x86_64.whl'.
My OS is CentOS7

import is ok but I came up with below message.

RuntimeError: Stream has unknown format: ..sample_gds/t10.gds in LayoutViewBase.load_layout

However I opened same gds file using Klayout, It was OK.
't10.gds' was downloaded from 'klayout/tree/master/testdata/gds'

Is there something to do with gds file?
Or have to convert 'gds' file to some proper 'format'?

---

The qtless-canvas branch now is in a fairly stable state. With the Python wheel available in the Artifacts section you should be able to do something like this without Qt:

import klayout.lay as lay
import klayout.db as db

lv = lay.LayoutView()
lv.load_layout("samples/lvs/ringo.gds", True)
lv.load_layer_props("samples/lvs/tech.lyp")
lv.set_config("background-color", "#300000")
lv.max_hier()

# signature same as get_image_with_options, but returns a PixelBuffer
# object which substitutes QImage in the non-Qt case:
pxbuf = lv.get_pixels_with_options(800, 500)
png_string = pxbuf.to_png_data()

with open("pixels.png", "wb") as file:
  file.write(png_string)

Explanation: The "lay" module contains a superset of the layout viewer/edtiro API. Specifically LayoutView is a non-Qt class now and offers much of the functionality of LayoutView. Except image export is done via a new class named PixelBuffer instead of QImage. The PNG read and write abilities come from libpng. It's also possible to turn an image into a byte string in PNG format as shown in the example above.

This however, is just the beginning. LayoutView offers a dynamic API which should basically be able to provide background drawing and mouse event simulation which eventually enables zoom, drawing of rulers and even editing the same way you are used from the desktop application. I'm currently evaluation ways how this can be utilized.

Matthias

Originally posted by @klayoutmatthias in #1082 (comment)

[klayoutmatthias]

That means that the stream reader plugins can't be found. Those are shared objects which a located in the deployment file hierarchy.

The master in WIP, so please don't expect everything to work out of the box. I'd recomment to try the latest master CI builds: https://github.com/KLayout/klayout/actions/runs/2834848667#artifacts

Otherwise, please build from source.

Matthias

[proppy]

I get a similar error with a fairly recent build: https://github.com/KLayout/klayout/actions/runs/3115315370, see:
https://colab.research.google.com/gist/proppy/ea45ad7edeb6338d2c71b31218f7f29b/klayout-playground.ipynb#scrollTo=uFHz1CpDUZuB

RuntimeError                              Traceback (most recent call last)
<ipython-input-23-9c4f86892a12> in <module>
      4 
      5 lv = lay.LayoutView()
----> 6 lv.load_layout('nand2.gds')
      7 lv.max_hier()
      8 pixels = lv.get_pixels_with_options(500, 500)

RuntimeError: Stream has unknown format: nand2.gds in LayoutViewBase.load_layout

I do see the plugin in the whl, so maybe there is an issue with the library search path?

  inflating: klayout/db_plugins/lib_gds2_dbpi.cpython-37m-x86_64-linux-gnu.so  
  inflating: klayout/db_plugins/lib_magic_dbpi.cpython-37m-x86_64-linux-gnu.so  
  inflating: klayout/db_plugins/lib_net_tracer_dbpi.cpython-37m-x86_64-linux-gnu.so  
  inflating: klayout/db_plugins/lib_pcb_dbpi.cpython-37m-x86_64-linux-gnu.so  
  inflating: klayout/db_plugins/lib_lefdef_dbpi.cpython-37m-x86_64-linux-gnu.so  
  inflating: klayout/db_plugins/lib_dxf_dbpi.cpython-37m-x86_64-linux-gnu.so  
  inflating: klayout/db_plugins/lib_oasis_dbpi.cpython-37m-x86_64-linux-gnu.so  
  inflating: klayout/db_plugins/lib_cif_dbpi.cpython-37m-x86_64-linux-gnu.so  

[proppy]

if I trace the dynamic loader attemps using:

%%bash -c 'LD_DEBUG=libs python'
import klayout.lay as lay
from IPython.display import Image

lv = lay.LayoutView()
lv.load_layout('nand2.gds')
lv.max_hier()
pixels = lv.get_pixels_with_options(500, 500)
Image(pixels.to_png_data(), width=500, height=500)

I get can't see it attempting to load anything from db_plugins: https://gist.github.com/proppy/1f6d8efae0b5cd75af1e91c428e7ae09

Looking at the RPATH from the different plugins, they seems to be referring to some other non-existing path (klayout.libs):

!objdump -x /usr/local/lib/python3.7/dist-packages/klayout/db_plugins/lib_gds2_dbpi.cpython-37m-x86_64-linux-gnu.so | grep -C 5 PATH
[6]
0s
!objdump -x /usr/local/lib/python3.7/dist-packages/klayout/db_plugins/lib_gds2_dbpi.cpython-37m-x86_64-linux-gnu.so | grep -C 5 PATH
         filesz 0x0000000000016c30 memsz 0x0000000000016c30 flags rw-
    NOTE off    0x0000000000098c08 vaddr 0x0000000000099c08 paddr 0x0000000000099c08 align 2**2
         filesz 0x0000000000000024 memsz 0x0000000000000024 flags r--

Dynamic Section:
  RPATH                $ORIGIN/..:$ORIGIN/../../klayout.libs
  NEEDED               lib_tl.cpython-37m-x86_64-linux-gnu.so
  NEEDED               lib_gsi.cpython-37m-x86_64-linux-gnu.so
  NEEDED               lib_db.cpython-37m-x86_64-linux-gnu.so
  NEEDED               libstdc++.so.6
  NEEDED               libm.so.6

I wonder if that's related to the comment in:
https://github.com/KLayout/klayout/blob/master/ci-scripts/docker/development_notes/fix_wheel.sh#L44

[proppy]

Interestingly the db module does seem to trigger the load of the gds plugin:

%%bash -c 'LD_DEBUG=libs python'
import klayout.db as db
ly = db.Layout()
ly.read('nand2.gds')
print(ly.layer_indexes())

https://gist.github.com/proppy/765d8d4d782d3aad172c122f8eb201cf

      2631:	calling init: /usr/local/lib/python3.7/dist-packages/klayout/db_plugins/lib_gds2_dbpi.cpython-37m-x86_64-linux-gnu.so
      2631:	
      2631:	/usr/local/lib/python3.7/dist-packages/klayout/db_plugins/lib_gds2_dbpi.cpython-37m-x86_64-linux-gnu.so: error: symbol lookup error: undefined symbol: dbp_init (fatal)
      2631:	

[klayoutmatthias]

Hi @proppy,

I just tried to reproduce the problem with the manylinux Python 3.10 wheel from the build mentioned above. But without success (or rather: without fail) :(

I used a fresh ubuntu:latest Docker container and system Python3/pip3.

I wonder how the RPATH can play a role at all, because when the db module loads the plugins all dependencies should already be satisfied by libraries already loaded. As you observed the db_plugins are not loaded by Python but by the stream engine inside klayout.db. It scans the "db_plugins" folder for shared objects and tries to load these.

Here is my LD_DEBUG output with "files":

      4994:	calling init: /usr/local/lib/python3.10/dist-packages/klayout/lib_db.cpython-310-x86_64-linux-gnu.so
      4994:	
      4994:	
      4994:	calling init: /usr/local/lib/python3.10/dist-packages/klayout/dbcore.cpython-310-x86_64-linux-gnu.so
      4994:	
      4994:	opening file=/usr/local/lib/python3.10/dist-packages/klayout/dbcore.cpython-310-x86_64-linux-gnu.so [0]; direct_opencount=1
      4994:	
      4994:	
--- db module starts loading the db_plugin modules dynamically ---
--- CIF format loading .. ---
      4994:	file=/usr/local/lib/python3.10/dist-packages/klayout/db_plugins/lib_cif_dbpi.cpython-310-x86_64-linux-gnu.so [0];  dynamically loaded by /usr/local/lib/python3.10/dist-packages/klayout/lib_db.cpython-310-x86_64-linux-gnu.so [0]
      4994:	file=/usr/local/lib/python3.10/dist-packages/klayout/db_plugins/lib_cif_dbpi.cpython-310-x86_64-linux-gnu.so [0];  generating link map
      4994:	  dynamic: 0x00007f148f1ee000  base: 0x00007f148f17d000   size: 0x0000000000082b18
      4994:	    entry: 0x00007f148f1a0540  phdr: 0x00007f148f17d040  phnum:                 10
      4994:	
      4994:	activating NODELETE for /usr/local/lib/python3.10/dist-packages/klayout/db_plugins/lib_cif_dbpi.cpython-310-x86_64-linux-gnu.so [0]
      4994:	
      4994:	calling init: /usr/local/lib/python3.10/dist-packages/klayout/db_plugins/lib_cif_dbpi.cpython-310-x86_64-linux-gnu.so
      4994:	
      4994:	opening file=/usr/local/lib/python3.10/dist-packages/klayout/db_plugins/lib_cif_dbpi.cpython-310-x86_64-linux-gnu.so [0]; direct_opencount=1
      4994:	
--- NOTE: this error is ignored: ---
      4994:	/usr/local/lib/python3.10/dist-packages/klayout/db_plugins/lib_cif_dbpi.cpython-310-x86_64-linux-gnu.so: error: symbol lookup error: undefined symbol: dbp_init (fatal)
      4994:	
--- DXF format loading .. ---
      4994:	file=/usr/local/lib/python3.10/dist-packages/klayout/db_plugins/lib_dxf_dbpi.cpython-310-x86_64-linux-gnu.so [0];  dynamically loaded by /usr/local/lib/python3.10/dist-packages/klayout/lib_db.cpython-310-x86_64-linux-gnu.so [0]
      4994:	file=/usr/local/lib/python3.10/dist-packages/klayout/db_plugins/lib_dxf_dbpi.cpython-310-x86_64-linux-gnu.so [0];  generating link map
...

So, no dependency resolution involved ...

Anyway, even then $ORIGIN/.. is included in the RPATH which should be sufficient. In my case, "ldd" does not show any unresolved dependencies when used in the "db_plugins" libraries.

I wonder how LD_DEBUG output continues in your case.

Matthias

[proppy]

I wonder how LD_DEBUG output continues in your case.

Here is some more LD_DEBUG=libs outputs:

• for 3.7: https://gist.github.com/proppy/43128c6ce97b385dfa939587b7bd4f11
• for 3.10: https://gist.github.com/proppy/f87eca22ed030e08400b180e424bc479

[klayoutmatthias]

Sorry, I lost this thread.
Is this still valid?

Matthias

[proppy]

@klayoutmatthias yep, I'm still getting this issue with the newly released klayout 0.28 and colab / python 3.8:

RuntimeError                              Traceback (most recent call last)
[<ipython-input-3-9ef2ce1918b7>](https://localhost:8080/#) in <module>
      3 
      4 lv = lay.LayoutView()
----> 5 lv.load_layout('gf180mcu_fd_sc_mcu9t5v0__addf_1.gds')
      6 lv.max_hier()
      7 pixels = lv.get_pixels_with_options(500, 500)

RuntimeError: Stream has unknown format: gf180mcu_fd_sc_mcu9t5v0__addf_1.gds in LayoutViewBase.load_layout

See https://colab.research.google.com/gist/proppy/8f44134519417f97538dc8e9b97cdfa8/klayout-0-28-load_layout-1146.ipynb

[proppy]

Looking at LD_DEBUG=libs debug output there doesn't seems to be anything loaded from db_plugins: https://gist.github.com/proppy/0cdacc0a37746965e04480dc0ecbddc6

[proppy]

Forcing the gds plugin to load with:

LD_PRELOAD=/usr/local/lib/python3.8/dist-packages/klayout/db_plugins/lib_gds2_dbpi.cpython-38-x86_64-linux-gnu.so 

does work around the issue:

%%bash -c 'LD_PRELOAD=/usr/local/lib/python3.8/dist-packages/klayout/db_plugins/lib_gds2_dbpi.cpython-38-x86_64-linux-gnu.so python'
import klayout.lay as lay
from IPython.display import Image

lv = lay.LayoutView()
lv.load_layout('gf180mcu_fd_sc_mcu9t5v0__addf_1.gds')
lv.max_hier()
pixels = lv.get_pixels_with_options(500, 500)
with open('gf180mcu_fd_sc_mcu9t5v0__addf_1.png', 'wb') as f:
  f.write(pixels.to_png_data())

[proppy]

Setting LD_LIBRARY_PATH to /usr/local/lib/python3.8/dist-packages/klayout/db_plugins/ doesn't seems to help, see:
https://gist.github.com/proppy/619ae23f5767b4cfe4f7293cd4839311

[proppy]

Found a cleaner workaround based on #1146 (comment):

• load the gds with klayout.db to force the gds plugin to be loaded
• load the gds again with klayout.lay reusing the plugin previously loaded

See:
https://colab.research.google.com/gist/proppy/7dac54b5aded7b05164791639d4bcae3/klayout-0-28-load_layout-1146.ipynb

[klayoutmatthias]

You're right.

But it should be enough do

import klayout.db

to enable the stream readers. It does not matter where in the code, you can safely import it always along with klayout.lay. import klayout.db properly initializes the DB library, while import klayout.lay just loads the shared object. I think this can be done implicitly too, but right now, that is missing.

So this should work too:

import klayout.db as db
import klayout.lay as lay

lv = lay.LayoutView()
lv.load_layout('gf180mcu_fd_sc_mcu9t5v0__addf_1.gds')
lv.max_hier()
pixels = lv.get_pixels_with_options(500, 500)
Image(pixels.to_png_data(), width=500, height=500)

Matthias

[proppy]

@klayoutmatthias confirmed just importing db also work, is that something we should document somewhere?

I'm also curious on how to associated the LayoutView with the Technology.

[proppy]

This seems to work assuming that the lyt reference the lyp appropriatly:

import klayout.db as db
import klayout.lay as lay

db.Technology.clear_technologies()
tech = db.Technology.create_technology('gf180mcu')
tech.load('gf180mcu.lyt')

lv = lay.LayoutView()
lv.load_layout('gf180mcu_fd_sc_mcu9t5v0__addf_1.gds')
lv.max_hier()
pixels = lv.get_pixels_with_options(500, 500)
with open('gf180mcu_fd_sc_mcu9t5v0__addf_1.png', 'wb') as f:
    f.write(pixels.to_png_data())

[proppy]

@klayoutmatthias I think we can close this :)

[klayoutmatthias]

Thanks :)